You are working on a project to replace a fraud detection legacy system. Your te

Question

You are working on a project to replace a fraud detection legacy system. Your team of 2 DS have access to a dataset of financial transactions comprised of 12 months of labeled historical data. The data contains several fields including the timestamp, card number, transaction location, amount, fraud score (from legacy system), and a flag indicating if the transaction was considered fraudulent or not. Your goal is to develop a predictive model that you can deploy in production with a detection rate above 70% while maintaining a low false positive rate of 1%.

How would you structure and plan the team’s work?

Based on the few fields exemplified above, which features would you suggest your team to try out?

The team has been working hard, but seems unable to go above 50% recall at 1% FPR. They can only achieve 70% recall at 1.5% FPR.

How would you handle the situation?

On a lucky strike, after the team introduced one new feature to the model, the performance bumped up to 95% recall at 1% FPR. There is now an increasingly high pressure from the project manager to deploy a model to production asap.

How would you advise the DS team?

How would you advise the project manager?

Explanation / Answer

In order to increase the detection rate above 70% with an FPR of 1% the focus must be on below parameters:

1. Card number: Card number is the most important parameter since it gives an idea about the card through which fraud has been conducted. Once card number is identified, it is easier to track the progress.

2. Transaction location: Transaction location is another important parameter which gives an idea about the location at which the fraud has been conducted. The security cameras at those locations can be monitored and fraud can be detected in a better way.

3. Fraud score: It is another important parameter which gives a hint about the level of fraud. Appropriate action can be taken after the fraud score has been identified.

DS team can be advised in the following ways:

1. Regular monitoring of the performance needs to be tracked on a regular basis.

2. Corrective action is to be taken if any deviation is measured from the regular process.

3. DS team training needs to be conducted on a regular basis so that they are aware of the new process and techniques.

4. Proper documentation and use of upgraded techniques to be followed on priority.

Project manager can be advised in below ways:

1. Checking of the whole process to be done regularly and suggestions to be given for improvement.

2. Guide the DS team on the process to be followed and help them if they are stuck anywhere.

3. Participate in decision making and change the process of there is any loophole after proper monitoring of the current process.

Related Questions

Navigate

• Browse (All)
• Browse Y
• Subjects

Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.