How to Evaluate Banking Software: A Pilot That Tests Real Operations

A banking software pilot should leave the buying team able to explain how a transaction works, who controls it and what happens when it goes wrong.

That requires more than a tour of the interface. A useful evaluation follows a complete business operation through the customer application, staff decisions, external providers and financial records. It produces evidence that product, engineering, operations and finance can assess together.

The framework below is Bastion Research's proposed approach to a focused pilot. It uses controlled test data and deliberately introduced failures to answer a practical question: can this configuration support the operation we intend to launch?

Start with one decision

Define what the pilot will help you decide. For example: can a new customer application and payment workflow operate with the institution's existing core?

Keep the first scope narrow enough to investigate properly. One entity, one customer type, one currency and one payment provider can reveal more than a broad demonstration in which every module receives a few minutes of attention.

Write down the journey, the systems involved, the people who will assess it and the evidence required. Also record which connections use a provider sandbox, which use a simulator and which are unavailable. A simulated rejection is useful evidence of application behaviour; it does not establish that the live provider connection is ready.

Agree the outcome before implementation starts. The decision might be to proceed with the evaluated scope, resolve specific gaps or reconsider the architecture. Each outcome should follow from recorded results.

Make ownership explicit

If the existing core will remain in place, the pilot must show how the new layer respects its authority. A cached balance on a new screen should have a defined relationship to the balance used to approve spending.

Create a small ownership map:

Where the core owns the balance, test reservation and posting through its supported interfaces. Where a new ledger is part of the scope, test that ledger directly. Avoid leaving both systems with independent authority to spend the same funds.

Follow one payment all the way through

Use a payment that needs an approval. Start with known account balances and an agreed fee rule, then create the beneficiary and payment instruction.

Follow the instruction through validation, any funds reservation, approval, provider submission, status updates and reconciliation. The precise sequence depends on the chosen product and provider. Record the expected sequence before running the test.

At each step, inspect what the customer sees, what the operator can do and what the financial records contain. A provider acknowledgement may confirm receipt without confirming settlement. Customer-facing labels should reflect that distinction.

Give the payment a traceable business reference. Connect it to approval events, provider references and accounting entries so that an operator can reconstruct the journey without asking a developer to search unrelated logs.

Finish by comparing the expected balance movement, fee and external result with the recorded outcome. Our article on ledger and reconciliation explains the relationship between these records in more detail.

Repeat the journey under failure conditions

Once the normal path is understood, introduce conditions that change the result or make it uncertain. The following matrix is a proposed acceptance test for an outgoing-payment pilot, adapted to the selected rail and provider contract.

For retries, agree what identifies the original intent and how long the provider honours that identity. Reusing an identifier with changed payment parameters should not silently create a different operation. AWS describes the underlying request-identity and parameter-validation principles in its guidance on idempotent APIs.

For notifications, use the actual provider's delivery contract. Stripe, for example, documents duplicate delivery and does not guarantee event order. That is a concrete reason to test these behaviours, rather than assume every callback arrives once and in sequence. See Stripe's webhook documentation.

A timeout deserves particular attention. It means the caller lacks a response; the remote outcome may still be successful. The pilot should demonstrate how the team resolves that uncertainty through an appropriate query, a safe retry or an operational investigation.

Let the operating team handle an exception

Ask an operator to investigate a delayed payment using the normal staff interface. Give them the access they would have in production.

Can they see the last confirmed event, the source of the status, the amount reserved or posted, and the next permitted action? Can they distinguish a customer cancellation request from a confirmed provider cancellation? Can they hand the case to another team without losing its history?

Then ask someone without the required permission to attempt the same sensitive action. Test the API as well as the visible button. OWASP recommends permission checks on every request; hiding a control in the interface does not establish authorisation. See the OWASP Authorisation Cheat Sheet.

Record any step that requires direct database editing or undocumented developer intervention. Such a step is an implementation task with an owner, not a completed operational workflow.

Measure a complete result

Agree acceptance criteria before measuring. A few useful measures are:

Choose performance thresholds from the intended service and its operating conditions. A lightly loaded sandbox result does not establish production capacity. A low average response time can also hide slow requests, so include an agreed percentile, such as the 95th percentile, and the error rate.

Financial correctness, required permissions and safe handling of uncertain outcomes should be decision gates. A weighted feature score can then help compare usability, implementation effort and coverage among configurations that pass those gates.

End with a decision record

Produce a short evidence pack containing the evaluated configuration, ownership map, scenario results, representative transaction traces and unresolved gaps. For every gap, identify its impact, owner and next verification step.

Keep production readiness as a separate, explicit decision. Capacity, operational coverage, security review, recovery arrangements and live-provider readiness may require work beyond a focused pilot. List what remains so that the pilot result can become a credible implementation plan.

The most useful outcome is a clear statement: this configuration supports these operations, under these tested conditions, with these remaining dependencies.