If crypto sounds limited or a bit old for what you want to achieve, you can refer to BouncyCastle, http://www.bouncycastle.org/csharp
It provides .NET and Java SDK and it is the most advanced open source library available out there.
You can find any sort of current crypto functions, authentication, digital signature, smartcard functionalities.
It doesn't come out of the box though, you must wrap the .NET classes in order to be called from PB as oleobject. Google for "Calling .NET Components from PowerBuilder" and you will find lots of article from Bruce Armstrong.
Eventually the new releases of PB will offer native support to .NET classes, stay tuned with the cool features that will show up
https://www.appeon.com/developers/roadmap
Best,
.m