Key information

  1. Status: Approved for delivery
  2. Reference: ST1386
  3. Version: 1.0
  4. Level: 5
  5. Typical duration to gateway: 24 months
  6. Typical EPA period: 4 months
  7. Maximum funding: £19000
  8. Route: Digital
  9. Date updated: 11/12/2023
  10. Approved for delivery: 11 December 2023
  11. Lars code: 746
  12. EQA provider: Ofqual
  13. Example progression routes:
  14. Review: this apprenticeship will be reviewed in accordance with our change request policy.
Print apprenticeship summary

Apprenticeship summary

Overview of the role

Build systems that collect, manage, and convert data into usable information for data scientists, data analysts and business intelligence analysts to interpret.

Occupation summary

This occupation is found in a wide range of public and private sector organisations who work with large data sets including Government departments, NHS, financial and professional services, IT companies, retail and sales and education providers.

The purpose of the occupation is to build systems that collect, manage, and convert data into usable information for data scientists, data analysts and business intelligence analysts to interpret. A data engineer’s main aim is to make data accessible and valid so that an organisation can use it to evaluate and optimise their performance. The role of the data engineer is pivotal to any organisation; it ensures that data pipelines are established to support data scientists and other business stakeholders.

A data engineer will build and implement data flows to connect operational systems, and re-engineer manual data flows to enable scalable and repeatable use. They integrate, support and manage the build of data streaming systems, writing extract transform and load scripts that perform in line with business requirements. 

They are responsible for providing high quality, transparent data that enables effective governance and smart business decisions. They will analyse the performance indicators of the data systems that provide clean, regular, and accurate data. A data engineer will understand how data and an organisation’s data architecture is essential to business outcomes.

A data engineer will be able to gather requirements for data solutions, and they demonstrate and articulate data solutions to stakeholders in a way that can be easily understood. Data engineering encompasses a range of activities from collecting data to employing various data processing frameworks, including but not limited to ETL (Extract, Transform, Load), and collaborating with data scientists and other data-centric roles. Data engineers may work in an office or work remotely depending on the sector they work in and location of the employer.

In their daily work, an employee in this occupation will work  autonomously or collaboratively with clients, in the business and or data team. A data engineer will work with data analysts, Data scientists and data architects and liaise with other teams and internal and external stakeholders to ensure their data requirements are captured and managed to the specified standard. They will also work closely with machine learning engineer (Ops), software engineers, software developers and technology teams.

An employee in this occupation will be responsible for completing their own work to specification, , ensuring they meet set deadlines. A data engineer contributes towards, engineering designs, plans, execution and evaluation working to time, cost and quality targets. They deliver to the product roadmap and are responsible for meeting quality requirements and working in accordance with health and safety and environmental considerations. They will work according to organisational procedures and policies, to maintain security and compliance.

Typical job titles include:

Data engineer

Duties

  • Duty 1 Build and optimise automated data systems and pipelines considering data quality, description, cataloguing, data cleaning, validation, technical documentation and requirements.
  • Duty 2 Integrate, support and manage data using standalone, distributed and cloud-based platforms. To ensure efficient, sustainable and effective provision of data storage solutions.
  • Duty 3 Support the identification and evaluation of opportunities for data acquisition and data enrichment.
  • Duty 4 Select and use appropriate tools to process data in any format, such as structured and unstructured data and in any mode of delivery, such as streaming or batching. Adapt to legacy systems as required.
  • Duty 5 Ensure resilience is built into data products against business continuity and disaster recovery plans, and document change management to limit service outages. Support and respond to incidents through the application of technology and service management best practice including configuration, change and incident management.
  • Duty 6 Analyse requirements, research scope and options and present recommendations for solutions to stakeholders.
  • Duty 7 Support the implementation of prototype or proof-of-concept data products within a production environment
  • Duty 8 Maintain data solutions as continually evolving products, to service the organisation, user or client requirements. Collaborate with technical support teams and stakeholders from implementation to management.
  • Duty 9 Working within compliance and contribute towards data governance, organisational policies, standards, and guidelines for data engineering.
  • Duty 10 Monitor the data system to meet service requirements to enable solutions such as data analysis, dashboards, data products, pipelines, and storage solutions.
  • Duty 11 Keep up to date with engineering developments to advance own skills and knowledge.

Apprenticeship summary

ST1386, data engineer level 5

This is a summary of the key things that you – the apprentice and your employer need to know about your end-point assessment (EPA). You and your employer should read the EPA plan for the full details. It has information on assessment method requirements, roles and responsibilities, and re-sits and re-takes.

What is an end-point assessment and why it happens

An EPA is an assessment at the end of your apprenticeship. It will assess you against the knowledge, skills, and behaviours (KSBs) in the occupational standard. Your training will cover the KSBs. The EPA is your opportunity to show an independent assessor how well you can carry out the occupation you have been trained for.

Your employer will choose an end-point assessment organisation (EPAO) to deliver the EPA. Your employer and training provider should tell you what to expect and how to prepare for your EPA.

The length of the training for this apprenticeship is typically 24 months. The EPA period is typically 4 months.

The overall grades available for this apprenticeship are:

  • fail
  • pass
  • merit
  • distinction

When you pass the EPA, you will be awarded your apprenticeship certificate.

EPA gateway

The EPA gateway is when the EPAO checks and confirms that you have met any requirements required before you start the EPA. You will only enter the gateway when your employer says you are ready.

The gateway requirements for your EPA are:

  • achieved English and mathematics qualifications in line with the apprenticeship funding rules
  • for the project evaluation report, presentation and questions, the project's title and scope must be agreed with the EPAO and a project summary submitted

Assessment methods


Project with report

You will complete a project and write a report. You will be asked to complete a project. The title and scope must be agreed with the EPAO at the gateway. The report should be a maximum of 3500 words (with a 10% tolerance).

You will have 10 weeks to complete the project and submit the report to the EPAO.

You need to prepare and give a presentation to an independent assessor. Your presentation slides and any supporting materials should be submitted at the same time as the project output. The presentation with questions will last at least 50 minutes. The independent assessor will ask at least 6 questions about the project and presentation.


Professional discussion

You will have a professional discussion with an independent assessor. It will last 80 minutes. They will ask you at least 10 questions. The questions will be about certain aspects of your occupation. You can use it to help answer the questions.

The EPAO will confirm where and when each assessment method will take place.

Who to contact for help or more information

You should speak to your employer if you have a query that relates to your job.

You should speak to your training provider if you have any questions about your training or EPA before it starts.

You should receive detailed information and support from the EPAO before the EPA starts. You should speak to them if you have any questions about your EPA once it has started.


Reasonable adjustments

If you have a disability, a physical or mental health condition or other special considerations, you may be able to have a reasonable adjustment that takes this into account. You should speak to your employer, training provider and EPAO and ask them what support you can get. The EPAO will decide if an adjustment is appropriate.

Print occupational standard

Details of the occupational standard

Occupation summary

This occupation is found in a wide range of public and private sector organisations who work with large data sets including Government departments, NHS, financial and professional services, IT companies, retail and sales and education providers.

The purpose of the occupation is to build systems that collect, manage, and convert data into usable information for data scientists, data analysts and business intelligence analysts to interpret. A data engineer’s main aim is to make data accessible and valid so that an organisation can use it to evaluate and optimise their performance. The role of the data engineer is pivotal to any organisation; it ensures that data pipelines are established to support data scientists and other business stakeholders.

A data engineer will build and implement data flows to connect operational systems, and re-engineer manual data flows to enable scalable and repeatable use. They integrate, support and manage the build of data streaming systems, writing extract transform and load scripts that perform in line with business requirements. 

They are responsible for providing high quality, transparent data that enables effective governance and smart business decisions. They will analyse the performance indicators of the data systems that provide clean, regular, and accurate data. A data engineer will understand how data and an organisation’s data architecture is essential to business outcomes.

A data engineer will be able to gather requirements for data solutions, and they demonstrate and articulate data solutions to stakeholders in a way that can be easily understood. Data engineering encompasses a range of activities from collecting data to employing various data processing frameworks, including but not limited to ETL (Extract, Transform, Load), and collaborating with data scientists and other data-centric roles. Data engineers may work in an office or work remotely depending on the sector they work in and location of the employer.

In their daily work, an employee in this occupation will work  autonomously or collaboratively with clients, in the business and or data team. A data engineer will work with data analysts, Data scientists and data architects and liaise with other teams and internal and external stakeholders to ensure their data requirements are captured and managed to the specified standard. They will also work closely with machine learning engineer (Ops), software engineers, software developers and technology teams.

An employee in this occupation will be responsible for completing their own work to specification, , ensuring they meet set deadlines. A data engineer contributes towards, engineering designs, plans, execution and evaluation working to time, cost and quality targets. They deliver to the product roadmap and are responsible for meeting quality requirements and working in accordance with health and safety and environmental considerations. They will work according to organisational procedures and policies, to maintain security and compliance.

Typical job titles include:

Data engineer

Occupation duties

Duty KSBs

Duty 1 Build and optimise automated data systems and pipelines considering data quality, description, cataloguing, data cleaning, validation, technical documentation and requirements.

K1 K2 K3 K4 K5 K6 K7 K8 K9 K10 K11 K12 K26

S1 S2 S3 S4 S5 S6 S7 S8 S13

B1 B2 B3

Duty 2 Integrate, support and manage data using standalone, distributed and cloud-based platforms. To ensure efficient, sustainable and effective provision of data storage solutions.

K3 K4 K10 K11 K13 K14 K15

S1 S2 S3 S7 S9 S10

B2 B4

Duty 3 Support the identification and evaluation of opportunities for data acquisition and data enrichment.

K10 K11 K16 K17

S11 S12

B2 B3

Duty 4 Select and use appropriate tools to process data in any format, such as structured and unstructured data and in any mode of delivery, such as streaming or batching. Adapt to legacy systems as required.

K18 K19 K20

S9 S14 S15 S16

B4

Duty 5 Ensure resilience is built into data products against business continuity and disaster recovery plans, and document change management to limit service outages. Support and respond to incidents through the application of technology and service management best practice including configuration, change and incident management.

K21 K22 K23

S10 S18 S19 S20 S21

B2 B3 B5

Duty 6 Analyse requirements, research scope and options and present recommendations for solutions to stakeholders.

K12 K30

S2 S3 S11 S22 S23 S27

B2 B3 B4

Duty 7 Support the implementation of prototype or proof-of-concept data products within a production environment

K6 K8 K24 K30

S17 S22 S24

B3 B4

Duty 8 Maintain data solutions as continually evolving products, to service the organisation, user or client requirements. Collaborate with technical support teams and stakeholders from implementation to management.

K6 K25 K30

S1 S2 S3 S11 S12 S21 S22 S25

B1 B3 B4

Duty 9 Working within compliance and contribute towards data governance, organisational policies, standards, and guidelines for data engineering.

K3 K4 K6 K10 K11

S5 S13 S17

B3

Duty 10 Monitor the data system to meet service requirements to enable solutions such as data analysis, dashboards, data products, pipelines, and storage solutions.

K27

S18 S19 S26

B3 B5

Duty 11 Keep up to date with engineering developments to advance own skills and knowledge.

K28 K29

S12 S28 S29

B6

KSBs

Knowledge

K1: Processes to monitor and optimise the performance of the availability, management and performance of data product. Back to Duty

K2: Methodologies for moving data from one system to another for storage and further handling. Back to Duty

K3: Data normalisation principles and the advantages they achieve in databases for data protection, redundancy, and inconsistent dependency. Back to Duty

K4: Frameworks for data quality, covering dimensions such as accuracy, completeness, consistency, timeliness, and accessibility. Back to Duty

K5: The inherent risks of data such as incomplete data, ethical data sources and how to ensure data quality. Back to Duty

K6: Software development principles for data products, including debugging, version control, and testing. Back to Duty

K7: Principles of sustainable data products and organisational responsibilities for environmental social governance. Back to Duty

K8: Deployment approaches for new data pipelines and automated processes. Back to Duty

K9: How to build a data product that complies with regulatory requirements. Back to Duty

K10: Concepts of data governance, including regulatory requirements, data privacy, security, and quality control. Legislation and its application to the safe use of data. Back to Duty

K11: Data and information security standards, ethical practices, policies and procedures relevant to data management activities such as data lineage and metadata management. Back to Duty

K12: How to cost and build a system whilst ensuring that organisational strategies for sustainable, net zero technologies are considered. Back to Duty

K13: The implications of financial, strategic and compliance regarding to security, scalability, compliance and cost of local, remote or distributed solutions. Back to Duty

K14: The uses of on-demand Cloud computing platform(s) in a public or private environment such as Amazon AWS, Google Cloud, Hadoop, IBM Cloud, Salesforce and Microsoft Azure. Back to Duty

K15: Data warehousing principles, including techniques such as star schemas, data lakes, and data marts. Back to Duty

K16: Principles of data, including open and public data, administrative data, and research data including the value of external data sources that can be used to enrich internal data. Examples of how business use direct data acquisition to support or augment business operations. Back to Duty

K17: Approaches to data integration and how combining disparate data sources delivers value to an organisation. Back to Duty

K18: How to use streaming, batching and on-demand services to move data from one location to another. Back to Duty

K19: Differences between structured, semi-structured, and unstructured data. Back to Duty

K20: Types and uses of data engineering tools and applications in own organisation. Back to Duty

K21: Policies and strategies to ensure business continuity for operations, particularly in relation to data provision. Back to Duty

K22: Technology and service management best practice including configuration, change and incident management. Back to Duty

K23: How to undertake analysis and root cause investigation. Back to Duty

K24: Processes for evaluating prototypes and taking them to implementation within a production environment. Back to Duty

K25: The lifecycle of implementing data solutions in a business, from scoping, though prototyping, development, production, and continuous improvement. Back to Duty

K26: Data development frameworks and approved organisational architectures. Back to Duty

K27: The principles of descriptive, predictive and prescriptive analytics. Back to Duty

K28: Continuous improvement including how to: capture good practice and lessons learned. Back to Duty

K29: Strategies for keeping up to date with new ways of working and technological developments in data science, data engineering and AI. Back to Duty

K30: The methods and techniques used to communicate messages to meet the needs of the audience. Back to Duty

Skills

S1: Collate, evaluate and refine user requirements to design the data product. Back to Duty

S2: Collate, evaluate and refine business requirements including cost, resourcing, and accessibility to design the data product. Back to Duty

S3: Design a data product to serve multiple needs and with scalability, efficiency, and security in mind. Back to Duty

S4: Automate data pipelines such as batch, real-time, on demand and other processes using programming languages and data integration platforms with graphical user interfaces. Back to Duty

S5: Produce and maintain technical documentation explaining the data product, that meets organisational, technical and non-technical user requirements, retaining critical information. Back to Duty

S6: Systematically clean, validate, and describe data at all stages of extract, transform, load (ETL). Back to Duty

S7: Work with different types of data stores, such as SQL, NoSQL, and distributed file system. Back to Duty

S8: Identify and troubleshoot issues with data processing pipelines. Back to Duty

S9: Query and manipulate data using tools and programming such as SQL and Python. Manage database access, and implement automated validation checks. Back to Duty

S10: Communicate downtime and issues with database access to stakeholders to mitigate the operational impact of unforeseen issues. Back to Duty

S11: Evaluate opportunities to extract value from existing data products through further development, considering costs, environmental impact and potential operational benefits. Back to Duty

S12: Maintain a working knowledge of data use cases within organisations. Back to Duty

S13: Use data systems securely to meet requirements and in line with organisational procedures and legislation. Back to Duty

S14: Identify new tools and technologies and recommend potential opportunities for use in own department or organisation. Back to Duty

S15: Optimise data ingestion processes by making use of appropriate data ingestion frameworks such as batch, streaming and on-demand. Back to Duty

S16: Develop algorithms and processes to extract structured data from unstructured sources. Back to Duty

S17: Apply and advocate for software development best practice when working with other data professionals throughout the business. Contribute to standards and ways of working that support software development principles. Back to Duty

S18: Develop simple forecasts and monitoring tools to anticipate or respond immediately to outages and incidents. Back to Duty

S19: Identify and escalate risks with suggested mitigation/resolutions as appropriate. Back to Duty

S20: Investigate and respond to incidents, identifying the root cause and resolution with internal and external stakeholders. Back to Duty

S21: Identify and remediate technical debt, assess for updates and obsolescence as part of continuous improvement. Back to Duty

S22: Develop, maintain collaborative relationships using adaptive business methodology with stakeholders such as, business users, data scientists, data analysts and business intelligence teams. Back to Duty

S23: Present, communicate, and disseminate messages about the data product, tailoring the message and medium to the needs of the audience. Back to Duty

S24: Evaluate the strengths and weaknesses of prototype data products and how these integrate within an organisation’s overarching data infrastructure. Back to Duty

S25: Assess and identify gaps in existing tools and technologies in respect of implementing changes required. Back to Duty

S26: Identify data quality metrics and track them to ensure the quality, accuracy and reliability of the data product. Back to Duty

S27: Selects and apply sustainable solutions to contribute to net zero and environmental strategies across the various stages of product and service delivery. Back to Duty

S28: Horizon scanning to identify new technologies that offer increased performance of data products. Back to Duty

S29: Implement personal strategies to keep up to date with new technology and ways of working. Back to Duty

Behaviours

B1: Acts proactively and takes accountability adapting positively to changing work priorities, ensuring deadlines are met. Back to Duty

B2: Works collaboratively with stakeholders and colleagues, developing strong working relationships to achieve common goals. Support an inclusive culture and treat technical and non- technical colleagues and stakeholders with respect. Back to Duty

B3: Quality focus that promotes continuous improvement utilising peer review techniques, innovation and creativity to the data system development process to improve processes and address business challenges. Back to Duty

B4: Takes personal responsibility towards net zero and prioritises environmental sustainability outcomes in how they carry out the duties of their role. Back to Duty

B5: Use initiative and innovation to problem solve and trouble shoot, providing creative solutions. Back to Duty

B6: Keeps abreast of developments in emerging, contemporary and advanced technologies to optimise sustainable data products and services. Back to Duty

Qualifications

English and Maths

Apprentices without level 2 English and maths will need to achieve this level prior to taking the End-Point Assessment. For those with an education, health and care plan or a legacy statement, the apprenticeship’s English and maths minimum requirement is Entry Level 3. A British Sign Language (BSL) qualification is an alternative to the English qualification for those whose primary language is BSL.

Print EPA plan

End-point assessment plan

V1.0

Introduction and overview

This document explains the requirements for end-point assessment (EPA) for the data engineer apprenticeship. End-point assessment organisations (EPAOs) must follow this when designing and delivering the EPA.

Data engineer apprentices, their employers and training providers should read this document.

A full-time data engineer apprentice typically spends 24 months on-programme (this means in training before the gateway). The apprentice must spend at least 12 months on-programme and complete the required amount of off-the-job training in line with the apprenticeship funding rules.

The apprentice must complete their training and meet the gateway requirements before starting their EPA. The EPA will assess occupational competence.

An approved EPAO must conduct the EPA for this apprenticeship. Employers must select an approved EPAO from the register of end-point assessment organisations (RoEPAO).

This EPA has 2 assessment methods.

The grades available for each assessment method are below.

Assessment method 1 - project evaluation report, presentation and questions:

  • fail

  • pass

  • distinction

Assessment method 2 - professional discussion:

  • fail

  • pass

  • distinction

The result from each assessment method is combined to decide the overall apprenticeship grade. The following grades are available for the apprenticeship:

  • fail

  • pass

  • merit

  • distinction

EPA summary table

On-programme - typically 24 months

The apprentice must:

  • complete training to develop the knowledge, skills and behaviours (KSBs) outlined in this degree-apprenticeship’s standard
  • complete training towards English and mathematics qualifications in line with the apprenticeship funding rules

End-point assessment gateway

The apprentice’s employer must be content that the apprentice has attained sufficient KSBs to complete the apprenticeship.

The apprentice must:

  • confirm they are ready to take the EPA
  • have achieved English and mathematics qualifications in line with the apprenticeship funding rules

For the project evaluation report, presentation and questions, the apprentice must submit a project brief. To ensure the project allows the apprentice to meet the KSBs mapped to this assessment method to the highest available grade, the EPAO should sign-off the project’s title and scope at the gateway to confirm it is suitable.

The apprentice must submit the gateway evidence to their EPAO, including any organisation specific policies and procedures requested by the EPAO.

End-point assessment - typically 4 months

The grades available for each assessment method are below

Project evaluation report, presentation and questions:

  • fail

  • pass

  • distinction

Professional discussion:

  • fail

  • pass

  • distinction

Overall EPA and apprenticeship can be graded:

    • fail
    • pass
    • merit
    • distinction

Duration of end-point assessment period

The EPA is taken in the EPA period. The EPA period starts when the EPAO confirms the gateway requirements have been met and is typically 4 months.

The EPAO should confirm the gateway requirements have been met and start the EPA as quickly as possible.

EPA gateway

The apprentice’s employer must be content that the apprentice has attained sufficient KSBs to complete the apprenticeship. The employer may take advice from the apprentice's training provider, but the employer must make the decision. The apprentice will then enter the gateway.

The apprentice must meet the gateway requirements before starting their EPA.

They must:

  • confirm they are ready to take the EPA
  • have achieved English and mathematics qualifications in line with the apprenticeship funding rules

  • submit a project brief for the project evaluation report, presentation and questions

  • confirm that they have completed a suitable project that meets the knowledge, skills and behaviours.

The apprentice must submit the gateway evidence to their EPAO, including any organisation specific policies and procedures requested by the EPAO.

Order of assessment methods

The assessment methods can be delivered in any order.

The result of one assessment method does not need to be known before starting the next.

Project evaluation report, presentation and questions

Overview

A project involves the apprentice completing a significant and defined piece of work that has a real business application and benefit. The project must meet the needs of the employer’s business and be relevant to the apprentice’s occupation and apprenticeship.

The agreed project will present a typical business task, appropriate for demonstrating the skills and knowledge on the standard. The agreed project will be comparable in terms of content and complexity for all apprentices - it is the context within which the knowledge, and skills must be demonstrated that will vary. The project is undertaken and completed on programme and pre-gateway to the EPA. The project itself is not part of the EPA. The project will typically be undertaken on the employer’s premises.

This assessment method has 2 components:

  • project with a project output

  • presentation with questions and answers

Together, these components give the apprentice the opportunity to demonstrate the KSBs mapped to this assessment method. They are assessed by an independent assessor.

Rationale

This assessment method is being used because:

A project evaluation report is the most valid method as it allows the demonstration of professional competence.

The project is based on a real life example of the apprentices’ everyday work in their industry. Therefore, ensuring that they can demonstrate the KSBs in practice.

Producing a project evaluation report and presentation reflects normal professional practice, so this assessment method is appropriate.

Apprentices are required to be concise and precise in their use of language in written and verbal communication.

The project evaluation report offers a realistic opportunity to combine project management, examples of data products and formal writing enabling the apprentice to reflect on approaches taken.

It is a holistic assessment method, allowing the apprentice to demonstrate KSBs in an integrated way.

By writing the evaluation report on the project and being questioned to understand rationale for choices made, risks and problems identified, resolutions and areas where further action could be required. This method will enable the apprentice to showcase their professional competency.

The project is completed before gateway and is not graded. The project evaluation report is assessed and must be completed after gateway.

Delivery

The apprentice must complete a project based on any of the following:

  • a specific problem
  • a recurring issue
  • an idea or opportunity

To ensure the project allows the apprentice to meet the KSBs mapped to this assessment method to the highest available grade, the EPAO must sign-off the project’s title and scope at the gateway to confirm it is suitable. The EPAO must refer to the grading descriptors to ensure that projects are pitched appropriately.

The project output must be in the form of a report and presentation.

The apprentice must start the project before gateway. The project evaluation report must be completed after gateway. The employer should ensure the apprentice has the time and resources, to plan and complete their project.

The apprentice may work as part of a team to complete the project, which could include internal colleagues or technical experts. The apprentice must however, complete their project report and presentation unaided and they must be reflective of their own role and contribution. The apprentice and their employer must confirm this when the report and any presentation materials are submitted.

Component 1: Project report

The report must include at least:

  • an executive summary (or abstract)
  • an introduction
  • the scope of the project (including key performance indicators, aims and objectives)
  • a project plan
  • research outcomes
  • data product outcomes
  • project outcomes
  • discussion of findings
  • recommendations and conclusions
  • references
  • appendix containing mapping of KSBs to the report.

The report must also include:

To ensure the project allows the apprentice to meet the KSBs mapped to this assessment method to the highest available grade, the EPAO should sign-off the project evaluation report's title and scope at the gateway to confirm it is suitable.

The project output must be in the form of an evaluation report.

The apprentice must start the project evaluation report after the gateway. They must complete and submit the report to the EPAO by the end of week 10 of the EPA period.

The employer should ensure the apprentice has the time and resources, within this period, to plan and complete their project evaluation report. The apprentice must complete their project evaluation report and the production of its components unaided.

The apprentice may work as part of a team to complete the project which could include technical internal or external support. However, the project evaluation report must be the apprentice’s own work and reflective of their own role and contribution. The apprentice and their employer must confirm that the project evaluation report is the apprentice’s own work when it is submitted.

The project report must have a word count of 3500 words. A tolerance of 10% above or below is allowed at the apprentice’s discretion. Appendices, references and diagrams are not included in this total. The apprentice must produce and include a mapping in an appendix, showing how the report evidences the KSBs mapped to this assessment method.

The apprentice must complete and submit the report and any presentation materials to the EPAO by the end of week 10 of the EPA period.

Component 2: Presentation with questions

The presentation with questions must be structured to give the apprentice the opportunity to demonstrate the KSBs mapped to this assessment method to the highest available grade.

The apprentice must prepare and deliver a presentation to an independent assessor. After the presentation, the independent assessor must ask the apprentice questions about their project, report and presentation.

The presentation should cover:

  • an overview of the project
  • the project scope (including key performance indicators)
  • summary of actions undertaken by the apprentice
  • project outcomes and how these were achieved

The presentation with questions must last 50 minutes. This will typically include a presentation of 20 minutes and questioning lasting 30 minutes. The independent assessor must use the full time available for questioning. The independent assessor can increase the time of the presentation and questioning by up to 10%. This time is to allow the apprentice to complete their last point or respond to a question if necessary.

The independent assessor must ask at least 6 questions. They must use the questions from the EPAO’s question bank or create their own questions in line with the EPAO’s training. Follow up questions are allowed where clarification is required.

The purpose of the independent assessor's questions is:

  • to verify that the activity was completed by the apprentice
  • to seek clarification where required
  • to assess those KSBs that the apprentice did not have the opportunity to demonstrate with the report, although these should be kept to a minimum
  • to assess level of competence against the grading descriptors

The apprentice must submit any presentation materials to the EPAO at the same time as the report - by the end of week 10 of the EPA period. The apprentice must notify the EPAO, at that point, of any technical requirements for the presentation.

During the presentation, the apprentice must have access to:

  • audio-visual presentation equipment
  • flip chart and writing and drawing materials
  • computer

The independent assessor must have at least 2 weeks to review the project report and any presentation materials, to allow them to prepare questions.

The apprentice must be given at least 2 weeks’ notice of the presentation with questions.

Assessment decision

The independent assessor must make the grading decision. They must assess the project components holistically when deciding the grade.

The independent assessor must keep accurate records of the assessment. They must record:

  • the KSBs demonstrated in the report and presentation with questions
  • the apprentice’s answers to questions
  • the grade achieved

Assessment location

The presentation with questions must take place in a suitable venue selected by the EPAO for example, the EPAO’s or employer’s premises. It should take place in a quiet room, free from distractions and influence.

The presentation with questions can be conducted by video conferencing. The EPAO must have processes in place to verify the identity of the apprentice and ensure the apprentice is not being aided.

Question and resource development

The EPAO must develop a purpose-built assessment specification and question bank. It is recommended this is done in consultation with employers of this occupation. The EPAO must maintain the security and confidentiality of EPA materials when consulting with employers. The assessment specification and question bank must be reviewed at least once a year to ensure they remain fit-for-purpose.

The assessment specification must be relevant to the occupation and demonstrate how to assess the KSBs mapped to this assessment method. The EPAO must ensure that questions are refined and developed to a high standard. The questions must be unpredictable. A question bank of sufficient size will support this.

The EPAO must ensure that the apprentice has a different set of questions in the case of re-sits or re-takes.

EPAO must produce the following materials to support the project:

  • independent assessor EPA materials which include:
    • training materials
    • administration materials
    • moderation and standardisation materials
    • guidance materials
    • grading guidance
    • question bank
  • EPA guidance for the apprentice and the employer

The EPAO must ensure that the EPA materials are subject to quality assurance procedures including standardisation and moderation.

Professional discussion

Overview

In the professional discussion, an independent assessor and apprentice have a formal two-way conversation. It gives the apprentice the opportunity to demonstrate the KSBs mapped to this assessment method.

Rationale

This assessment method is being used because:

It provides the apprentice with the opportunity to discuss and show case their depth of understanding the knowledge, skills and behaviours that may not naturally occur as part of the project.

It allows the independent assessor to consider the context and sector that the apprentice operates within, giving flexibility to ensure that all the KSBs can be assessed appropriately.

The professional discussion is cost effective, and it allows consideration of the potential need to conduct the EPA remotely.

Delivery

The professional discussion must be structured to give the apprentice the opportunity to demonstrate the KSBs mapped to this assessment method to the highest available grade.

An independent assessor must conduct and assess the professional discussion.

The apprentice may choose to end the assessment method early. The apprentice must be confident they have demonstrated competence against the assessment requirements for the assessment method. The independent assessor or EPAO must ensure the apprentice is fully aware of all assessment requirements. The independent assessor or EPAO cannot suggest or choose to end any assessment methods early unless in an emergency. The EPAO is responsible for ensuring the apprentice understands the implications of ending an assessment early if they choose to do so. The independent assessor may suggest the assessment continues. The independent assessor must document the apprentice’s request to end the assessment early.

The EPAO will ask ten questions, two for each of the themes:

  • Data quality and performance
  • Problem solving
  • Regulatory Compliance
  • Continuous Improvement
  • Continuous professional development

The EPAO must give an apprentice 2 weeks' notice of the professional discussion.

The professional discussion must last for 80 minutes. The independent assessor can increase the time of the professional discussion by up to 10%. This time is to allow the apprentice to respond to a question if necessary.

The independent assessor must ask at least 10 questions. The independent assessor must use the questions from the EPAO’s question bank or create their own questions in line with the EPAO’s training. Follow-up questions are allowed where clarification is required.

The independent assessor must make the grading decision.

The independent assessor must keep accurate records of the assessment. They must record:

  • the apprentice’s answers to questions
  • the KSBs demonstrated in answers to questions
  • the grade achieved 

Assessment location

The professional discussion must take place in a suitable venue selected by the EPAO for example, the EPAO’s or employer’s premises.

The professional discussion can be conducted by video conferencing. The EPAO must have processes in place to verify the identity of the apprentice and ensure the apprentice is not being aided.

The professional discussion should take place in a quiet room, free from distractions and influence.

Question and resource development

The EPAO must develop a purpose-built assessment specification and question bank. It is recommended this is done in consultation with employers of this occupation. The EPAO must maintain the security and confidentiality of EPA materials when consulting with employers. The assessment specification and question bank must be reviewed at least once a year to ensure they remain fit-for-purpose.

The assessment specification must be relevant to the occupation and demonstrate how to assess the KSBs mapped to this assessment method. The EPAO must ensure that questions are refined and developed to a high standard. The questions must be unpredictable. A question bank of sufficient size will support this.

The EPAO must ensure that the apprentice has a different set of questions in the case of re-sits or re-takes.

The EPAO must produce the following materials to support the professional discussion:

  • independent assessor assessment materials which include:
    • training materials
    • administration materials
    • moderation and standardisation materials
    • guidance materials
    • grading guidance
    • question bank
  • EPA guidance for the apprentice and the employer

The EPAO must ensure that the EPA materials are subject to quality assurance procedures including standardisation and moderation.

Grading

Project evaluation report, presentation and questions

Theme
KSBs
Pass
Apprentices must demonstrate all of the pass descriptors
Distinction
Apprentices must demonstrate all of the pass descriptors and all of the distinction descriptors
Data product design
K6 K7 K9 K12 K13 K14 S1 S2 S3 S4 S5 S27 B1

Demonstrates how they have collated, evaluated and refined user requirements to design and build a scalable data product that serves multiple needs and complies with regulatory requirements. (K9, S1, S3)

Explains how they collated, evaluated and refined business requirements, to design, build and maintain a system whilst ensuring that organisational strategies for sustainable, net-zero technologies are considered. (K12 & S2)

Explains how they selected sustainable solutions in relation to data products and environmental social governance to ensure the use of less carbon across the various stages of product and service delivery. (K7, S27)

Demonstrates how they used security, scalability and governance when automating data pipelines using programming languages and data integration platforms with graphical user interfaces. (K13, S4)

Demonstrates how they have taken accountability produced and maintained technical documentation for a data product in order to meet organisational user requirements, whilst adapting to changing work priorities to ensure that deadlines are met. (S5, B1)

Explains how debugging, version control and testing have an impact on  software development and the principles for data products. (K6)

Outlines the uses of different on-demand cloud computing platforms. (K14)

Justifies how the data product created met the requirements and served multiple needs (S1, S3)

Data product deployment and evaluation
K2 K4 K8 K15 K17 K19 K20 K24 K25 K26 S6 S9 S16 S24 S26

Explains the deployment approaches processes for new data pipelines and automated processes.(K8)

Explains techniques such as star schemas, data lakes and data marts and the impact they have on data warehousing principles. (K15)

Demonstrate how to systematically clean, validate and describe data at all stages of extract, transform and load, showing how combining disparate data sources and taking different approaches to data integration delivers value to an organisation.  (K17, S6)

Describes the types and uses of data engineering tools in their own organisation and how they apply them. (K20)

Evaluates the strengths and weaknesses of prototype data products to integrate within an organisation’s overarching data structure, taking into consideration the lifecycle of implementing data solutions in a business. (K24, K25, S24)

Describes the approved organisational architectures and the relevant data development frameworks. (K26)

Identifies data quality metrics and their frameworks and tracks them to ensure quality, accuracy and reliability of the data product. (K4, S26)

Demonstrates the use of tools and programming to query and manipulate data and implement automated validation checks, showing the methodologies used for moving data from one system to another for storage and handling. (K2, S9)

Explains how they have worked with structured, semi-structured and unstructured data, developing algorithms to extract from sources (K19, S16)

Evaluates the success of the algorithm developed (S16)

Collaborative working
K30 S22 S23 B2

Outlines the methods and techniques used to communicate messages about the data product that meet the needs of the audience. (K30, S23)

Explains how they worked collaboratively with different technical and non-technical stakeholders, using adaptive business methodology to support an inclusive culture and develop and maintain strong working relationships in order to achieve common goals. (S22, B2)

Evaluate the impact of the methods and techniques used to communicate messages about the data product to the audience. (K30, S23)

Professional discussion

Theme
KSBs
Pass
Apprentices must demonstrate all of the pass descriptors
Distinction
Apprentices must demonstrate all of the pass descriptors and all of the distinction descriptors
Data quality and performance
K1 K3 K5 K18 K27 S7 S15

 

Explains how they monitor different types of data store to optimise system management, performance and availability. (K1, S7)

Defines data normalisation principles and the advantages that they achieve for data protection, redundancy and inconsistent dependency. (K3)

Explains the inherent risks of data and how to ensure data quality (K5)

Explains the principles of descriptive, predictive and prescriptive analytics. (K27)

Describes how they use data ingestion frameworks such as streaming, batching and on demand services to move data from one location to another in order to optimise data ingestion processes. (K18, S15)

Compares and contrasts the different types of data stores they have used and how they optimised performance (K1, S7)

Problem Solving
K21 K22 K23 S8 S10 S12 S18 S19 S20 B5

Describes technology and service management best practice. (K22)

Explains how they identify and escalate risks and incidents, communicating downtime and issues with database access in line with policies in order to mitigate operational impact whilst ensuring business continuity. (K21, S10, S18, S19)

Explains how they have maintained a working knowledge of data use cases within organisations. (S12)

Explains how their analysis of root cause investigation is used to respond to incidents within data processing pipelines, whilst troubleshooting and providing resolutions to stakeholders. (K23, S8, S20, B5)

 

Justifies the approach taken to manage risks and incidents to maintain business continuity. (S18, S19)

Regulatory Compliance
K10 K11 S13

Explains their use of data, information security standards, ethical practices and data management policies and procedures to ensure data systems are used securely and in accordance with relevant legislation. (K11, S13)

Explains the legislative associated with the use and collation of data, including concepts of data governance and regulatory requirements. (K10)

None

Continuous Improvement
K16 K28 S11 S14 S17 S21 S25 S28 B3 B4

Outlines how they evaluate opportunities to extract value from existing data products whilst applying the principles of data and considering costs, environmental impact and potential operating benefits. (K16, S11)

Explains how they take personal responsibility within the duties of their role to identify new tools and technologies, and recommend potential opportunities for use in own department or organisation in order to prioritise environmental sustainability outcomes to work towards net zero. (S14, B4)

 

Explains how they take a quality focussed approach to identify and remediate technical debt and assess for updates and obsolescence within their promotion of continuous improvement, by utilising peer review techniques and capturing good practice, to provide innovation and creativity to the data system development process in order to improve processes and address business challenges. (K28, S21, B3)

Explains how they apply ways of working that support software development principles and advocate software development best practice when working with other data professionals. (S17)

Explains how they identify and assess new technologies, as well as gaps in existing tools and technologies, that offer increased performance of data products and implementation of changes required. (S25, S28)

Evaluates the impact that the implementation of identified new technologies would have on practices within the organisation. (S25, S28)

Continuous professional development
K29 S29 B6

 

Explains how they have implemented personal strategies for keeping up to date with new ways of working and to keep abreast of developments in emerging, contemporary and advanced technologies, in order to keep up to date with new technologies and technological developments in data science, data engineering and AI and to optimise sustainable products and services (K29, S29, B6)

Evaluate the impact that keeping up to date with technological developments has had on their own professional development. (S29) 

Overall EPA grading

Performance in the EPA determines the overall grade of:

  • fail

  • pass

  • merit

  • distinction

An independent assessor must individually grade the project evaluation report, presentation and questions and professional discussion in line with this EPA plan.

The EPAO must combine the individual assessment method grades to determine the overall EPA grade.

If the apprentice fails one assessment method or more, they will be awarded an overall fail.

To achieve an overall pass, the apprentice must achieve at least a pass in all the assessment methods. Both assessment methods are weighted equally in their contribution to the overall EPA grade.

Grades from individual assessment methods must be combined in the following way to determine the grade of the EPA overall.

Project evaluation report, presentation and questions Professional discussion Overall Grading
Fail Fail Fail
Fail Pass Fail
Pass Fail Fail
Pass Pass Pass
Pass Distinction Merit
Distinction Pass Merit
Distinction Distinction Distinction

Re-sits and re-takes

If the apprentice fails one assessment method or more, they can take a re-sit or a re-take at their employer’s discretion. The apprentice’s employer needs to agree that a re-sit or re-take is appropriate. A re-sit does not need further learning, whereas a re-take does. The apprentice should have a supportive action plan to prepare for a re-sit or a re-take.

The employer and the EPAO should agree the timescale for a re-sit or re-take. A re-sit is typically taken within 3 months of the EPA outcome notification. The timescale for a re-take is dependent on how much re-training is required and is typically taken within 6 months of the EPA outcome notification.

If the apprentice fails the project assessment method, they must amend the project output in line with the independent assessor’s feedback. The apprentice will be given 4 weeks to rework and submit the amended report.

Failed assessment methods must be re-sat or re-taken within a 6-month period from the EPA outcome notification, otherwise the entire EPA will need to be re-sat or re-taken in full.

Re-sits and re-takes are not offered to an apprentice wishing to move from pass to a higher grade.

The apprentice will get a maximum EPA grade of pass for a re-sit or re-take, unless the EPAO determines there are exceptional circumstances.

Roles and responsibilities

Roles Responsibilities

Apprentice

As a minimum, the apprentice should:

  • complete on-programme training to meet the KSBs as outlined in the apprenticeship standard for a minimum of 12 months
  • complete the required amount of off-the-job training specified by the apprenticeship funding rules and as arranged by the employer and training provider
  • understand the purpose and importance of EPA
  • prepare for and undertake the EPA including meeting all gateway requirements
  • ensure that all supporting evidence required at the gateway is submitted in line with this EPA plan

Employer

As a minimum, the apprentice's employer must:

  • select the EPAO and training provider
  • work with the training provider (where applicable) to support the apprentice in the workplace and to provide the opportunities for the apprentice to develop the KSBs
  • arrange and support off-the-job training to be undertaken by the apprentice 
  • decide when the apprentice is working at or above the apprenticeship standard and is ready for EPA
  • ensure the apprentice is prepared for the EPA
  • ensure that all supporting evidence required at the gateway is submitted in line with this EPA plan
  • confirm arrangements with the EPAO for the EPA (who, when, where) in a timely manner
  • provide the EPAO with access to any employer-specific documentation as required for example, company policies
  • ensure that the EPA is scheduled with the EPAO for a date and time which allows appropriate opportunity for the apprentice to meet the KSBs
  • ensure the apprentice is given sufficient time away from regular duties to prepare for, and complete the EPA
  • ensure that any required supervision during the EPA period, as stated within this EPA plan, is in place
  • ensure the apprentice has access to the resources used to fulfil their role and carry out the EPA for workplace based assessments
  • remain independent from the delivery of the EPA
  • pass the certificate to the apprentice upon receipt

EPAO

As a minimum, the EPAO must:

  • conform to the requirements of this EPA plan and deliver its requirements in a timely manner
  • conform to the requirements of the RoEPAO
  • conform to the requirements of the external quality assurance provider (EQAP)
  • understand the apprenticeship including the apprenticeship standard and, EPA plan
  • make all necessary contractual arrangements including agreeing the price of the EPA
  • develop and produce assessment materials including specifications and marking materials (for example mark schemes, practice materials, training material)
  • maintain and apply a policy for the declaration and management of conflict of interests and independence. This must ensure, as a minimum, there is no personal benefit or detriment for those delivering the EPA or from the result of an assessment. It must cover:
    • apprentices
    • employers
    • independent assessors
    • any other roles involved in delivery or grading of the EPA
  • have quality assurance systems and procedures that ensure fair, reliable and consistent assessment and maintain records of internal quality assurance (IQA) activity for external quality assurance (EQA) purposes
  • appoint independent, competent, and suitably qualified assessors in line with the requirements of this EPA plan
  • appoint administrators, invigilators and any other roles where required to facilitate the EPA
  • deliver induction, initial and on-going training for all their independent assessors and any other roles involved in the delivery or grading of the EPA as specified within this EPA plan. This should include how to record the rationale and evidence for grading decisions where required
  • conduct standardisation with all their independent assessors before allowing them to deliver an EPA, when the EPA is updated, and at least once a year
  • conduct moderation across all of their independent assessors decisions once EPAs have started according to a sampling plan, with associated risk rating of independent assessors
  • monitor the performance of all their independent assessors and provide additional training where necessary
  • develop and provide assessment recording documentation to ensure a clear and auditable process is in place for providing assessment decisions and feedback to all relevant stakeholders
  • use language in the development and delivery of the EPA that is appropriate to the level of the apprenticeship
  • arrange for the EPA to take place in a timely manner, in consultation with the employer
  • provide information, advice, and guidance documentation to enable apprentices, employers and training providers to prepare for the EPA
  • confirm the gateway requirements have been met before they start the EPA for an apprentice
  • host the EPA or make suitable alternative arrangements
  • maintain the security of the EPA including, but not limited to, verifying the identity of the apprentice, invigilation and security of materials
  • where the EPA plan permits assessment away from the workplace, ensure that the apprentice has access to the required resources and liaise with the employer to agree this if necessary
  • confirm overall grade awarded
  • maintain and apply a policy for conducting appeals

Independent assessor

As a minimum, an independent assessor must:

  • be independent, with no conflict of interest with the apprentice, their employer or training provider, specifically, they must not receive a personal benefit or detriment from the result of the assessment
  • have, maintain and be able to evidence up-to-date knowledge and expertise of the occupation
  • have the competence to assess the EPA and meet the requirements of the IQA section of this EPA plan
  • understand the apprenticeship’s occupational standard and EPA plan
  • attend induction and standardisation events before they conduct an EPA for the first time, when the EPA is updated, and at least once a year
  • use language in the delivery of the EPA that is appropriate to the level of the apprenticeship
  • work with other personnel, where used, in the preparation and delivery of assessment methods
  • conduct the EPA to assess the apprentice against the KSBs and in line with the EPA plan
  • make final grading decisions in line with this EPA plan
  • record and report assessment outcome decisions
  • comply with the IQA requirements of the EPAO
  • comply with external quality assurance (EQA) requirements

Training provider

As a minimum, the training provider must:

  • conform to the requirements of the register of apprenticeship training providers (RoATP)
  • ensure procedures are in place to mitigate against any conflict of interest
  • work with the employer and support the apprentice during the off-the-job training to provide the opportunities to develop the KSBs as outlined in the occupational standard
  • deliver training to the apprentice as outlined in their apprenticeship agreement
  • monitor the apprentice’s progress during any training provider led on-programme learning
  • ensure the apprentice is prepared for the EPA
  • advise the employer, upon request, on the apprentice’s readiness for EPA
  • ensure that all supporting evidence required at the gateway is submitted in line with this EPA plan
  • remain independent from the delivery of the EPA

Reasonable adjustments

The EPAO must have reasonable adjustments arrangements for the EPA.

This should include:

  • how an apprentice qualifies for reasonable adjustment
  • what reasonable adjustments may be made

Adjustments must maintain the validity, reliability and integrity of the EPA as outlined in this EPA plan.

Internal quality assurance

Internal quality assurance refers to the strategies, policies and procedures that an EPAO must have in place to ensure valid, consistent and reliable EPA decisions.

EPAOs for this EPA must adhere to the requirements within the roles and responsibilities table.

They must also appoint independent assessors who:

  • have recent relevant experience of the occupation or sector to at least occupational level 6 gained in the last 5 years or significant experience of the occupation or sector

Value for money

Affordability of the EPA will be aided by using at least some of the following:

  • completing applicable assessment methods online (for example computer-based assessment)
  • using the employer’s premises
  • conducting assessment methods on the same day

Professional recognition

This apprenticeship is not aligned to professional recognition.

KSB mapping table

Knowledge Assessment methods
K1

Processes to monitor and optimise the performance of the availability, management and performance of data product.

Back to Grading
Professional discussion
K2

Methodologies for moving data from one system to another for storage and further handling.

Back to Grading
Project evaluation report, presentation and questions
K3

Data normalisation principles and the advantages they achieve in databases for data protection, redundancy, and inconsistent dependency.

Back to Grading
Professional discussion
K4

Frameworks for data quality, covering dimensions such as accuracy, completeness, consistency, timeliness, and accessibility.

Back to Grading
Project evaluation report, presentation and questions
K5

The inherent risks of data such as incomplete data, ethical data sources and how to ensure data quality.

Back to Grading
Professional discussion
K6

Software development principles for data products, including debugging, version control, and testing.

Back to Grading
Project evaluation report, presentation and questions
K7

Principles of sustainable data products and organisational responsibilities for environmental social governance.

Back to Grading
Project evaluation report, presentation and questions
K8

Deployment approaches for new data pipelines and automated processes.

Back to Grading
Project evaluation report, presentation and questions
K9

How to build a data product that complies with regulatory requirements.

Back to Grading
Project evaluation report, presentation and questions
K10

Concepts of data governance, including regulatory requirements, data privacy, security, and quality control. Legislation and its application to the safe use of data.

Back to Grading
Professional discussion
K11

Data and information security standards, ethical practices, policies and procedures relevant to data management activities such as data lineage and metadata management.

Back to Grading
Professional discussion
K12

How to cost and build a system whilst ensuring that organisational strategies for sustainable, net zero technologies are considered.

Back to Grading
Project evaluation report, presentation and questions
K13

The implications of financial, strategic and compliance regarding to security, scalability, compliance and cost of local, remote or distributed solutions.

Back to Grading
Project evaluation report, presentation and questions
K14

The uses of on-demand Cloud computing platform(s) in a public or private environment such as Amazon AWS, Google Cloud, Hadoop, IBM Cloud, Salesforce and Microsoft Azure.

Back to Grading
Project evaluation report, presentation and questions
K15

Data warehousing principles, including techniques such as star schemas, data lakes, and data marts.

Back to Grading
Project evaluation report, presentation and questions
K16

Principles of data, including open and public data, administrative data, and research data including the value of external data sources that can be used to enrich internal data. Examples of how business use direct data acquisition to support or augment business operations.

Back to Grading
Professional discussion
K17

Approaches to data integration and how combining disparate data sources delivers value to an organisation.

Back to Grading
Project evaluation report, presentation and questions
K18

How to use streaming, batching and on-demand services to move data from one location to another.

Back to Grading
Professional discussion
K19

Differences between structured, semi-structured, and unstructured data.

Back to Grading
Project evaluation report, presentation and questions
K20

Types and uses of data engineering tools and applications in own organisation.

Back to Grading
Project evaluation report, presentation and questions
K21

Policies and strategies to ensure business continuity for operations, particularly in relation to data provision.

Back to Grading
Professional discussion
K22

Technology and service management best practice including configuration, change and incident management.

Back to Grading
Professional discussion
K23

How to undertake analysis and root cause investigation.

Back to Grading
Professional discussion
K24

Processes for evaluating prototypes and taking them to implementation within a production environment.

Back to Grading
Project evaluation report, presentation and questions
K25

The lifecycle of implementing data solutions in a business, from scoping, though prototyping, development, production, and continuous improvement.

Back to Grading
Project evaluation report, presentation and questions
K26

Data development frameworks and approved organisational architectures.

Back to Grading
Project evaluation report, presentation and questions
K27

The principles of descriptive, predictive and prescriptive analytics.

Back to Grading
Professional discussion
K28

Continuous improvement including how to: capture good practice and lessons learned.

Back to Grading
Professional discussion
K29

Strategies for keeping up to date with new ways of working and technological developments in data science, data engineering and AI.

Back to Grading
Professional discussion
K30

The methods and techniques used to communicate messages to meet the needs of the audience.

Back to Grading
Project evaluation report, presentation and questions
Skill Assessment methods
S1

Collate, evaluate and refine user requirements to design the data product.

Back to Grading
Project evaluation report, presentation and questions
S2

Collate, evaluate and refine business requirements including cost, resourcing, and accessibility to design the data product.

Back to Grading
Project evaluation report, presentation and questions
S3

Design a data product to serve multiple needs and with scalability, efficiency, and security in mind.

Back to Grading
Project evaluation report, presentation and questions
S4

Automate data pipelines such as batch, real-time, on demand and other processes using programming languages and data integration platforms with graphical user interfaces.

Back to Grading
Project evaluation report, presentation and questions
S5

Produce and maintain technical documentation explaining the data product, that meets organisational, technical and non-technical user requirements, retaining critical information.

Back to Grading
Project evaluation report, presentation and questions
S6

Systematically clean, validate, and describe data at all stages of extract, transform, load (ETL).

Back to Grading
Project evaluation report, presentation and questions
S7

Work with different types of data stores, such as SQL, NoSQL, and distributed file system.

Back to Grading
Professional discussion
S8

Identify and troubleshoot issues with data processing pipelines.

Back to Grading
Professional discussion
S9

Query and manipulate data using tools and programming such as SQL and Python. Manage database access, and implement automated validation checks.

Back to Grading
Project evaluation report, presentation and questions
S10

Communicate downtime and issues with database access to stakeholders to mitigate the operational impact of unforeseen issues.

Back to Grading
Professional discussion
S11

Evaluate opportunities to extract value from existing data products through further development, considering costs, environmental impact and potential operational benefits.

Back to Grading
Professional discussion
S12

Maintain a working knowledge of data use cases within organisations.

Back to Grading
Professional discussion
S13

Use data systems securely to meet requirements and in line with organisational procedures and legislation.

Back to Grading
Professional discussion
S14

Identify new tools and technologies and recommend potential opportunities for use in own department or organisation.

Back to Grading
Professional discussion
S15

Optimise data ingestion processes by making use of appropriate data ingestion frameworks such as batch, streaming and on-demand.

Back to Grading
Professional discussion
S16

Develop algorithms and processes to extract structured data from unstructured sources.

Back to Grading
Project evaluation report, presentation and questions
S17

Apply and advocate for software development best practice when working with other data professionals throughout the business. Contribute to standards and ways of working that support software development principles.

Back to Grading
Professional discussion
S18

Develop simple forecasts and monitoring tools to anticipate or respond immediately to outages and incidents.

Back to Grading
Professional discussion
S19

Identify and escalate risks with suggested mitigation/resolutions as appropriate.

Back to Grading
Professional discussion
S20

Investigate and respond to incidents, identifying the root cause and resolution with internal and external stakeholders.

Back to Grading
Professional discussion
S21

Identify and remediate technical debt, assess for updates and obsolescence as part of continuous improvement.

Back to Grading
Professional discussion
S22

Develop, maintain collaborative relationships using adaptive business methodology with stakeholders such as, business users, data scientists, data analysts and business intelligence teams.

Back to Grading
Project evaluation report, presentation and questions
S23

Present, communicate, and disseminate messages about the data product, tailoring the message and medium to the needs of the audience.

Back to Grading
Project evaluation report, presentation and questions
S24

Evaluate the strengths and weaknesses of prototype data products and how these integrate within an organisation’s overarching data infrastructure.

Back to Grading
Project evaluation report, presentation and questions
S25

Assess and identify gaps in existing tools and technologies in respect of implementing changes required.

Back to Grading
Professional discussion
S26

Identify data quality metrics and track them to ensure the quality, accuracy and reliability of the data product.

Back to Grading
Project evaluation report, presentation and questions
S27

Selects and apply sustainable solutions to contribute to net zero and environmental strategies across the various stages of product and service delivery.

Back to Grading
Project evaluation report, presentation and questions
S28

Horizon scanning to identify new technologies that offer increased performance of data products.

Back to Grading
Professional discussion
S29

Implement personal strategies to keep up to date with new technology and ways of working.

Back to Grading
Professional discussion
Behaviour Assessment methods
B1

Acts proactively and takes accountability adapting positively to changing work priorities, ensuring deadlines are met.

Back to Grading
Project evaluation report, presentation and questions
B2

Works collaboratively with stakeholders and colleagues, developing strong working relationships to achieve common goals. Support an inclusive culture and treat technical and non- technical colleagues and stakeholders with respect.

Back to Grading
Project evaluation report, presentation and questions
B3

Quality focus that promotes continuous improvement utilising peer review techniques, innovation and creativity to the data system development process to improve processes and address business challenges.

Back to Grading
Professional discussion
B4

Takes personal responsibility towards net zero and prioritises environmental sustainability outcomes in how they carry out the duties of their role.

Back to Grading
Professional discussion
B5

Use initiative and innovation to problem solve and trouble shoot, providing creative solutions.

Back to Grading
Professional discussion
B6

Keeps abreast of developments in emerging, contemporary and advanced technologies to optimise sustainable data products and services.

Back to Grading
Professional discussion

Mapping of KSBs to grade themes

Project evaluation report, presentation and questions

KSBS GROUPED BY THEME Knowledge Skills Behaviour
Data product design
K6 K7 K9 K12 K13 K14
S1 S2 S3 S4 S5 S27
B1

Software development principles for data products, including debugging, version control, and testing. (K6)

Principles of sustainable data products and organisational responsibilities for environmental social governance. (K7)

How to build a data product that complies with regulatory requirements. (K9)

How to cost and build a system whilst ensuring that organisational strategies for sustainable, net zero technologies are considered. (K12)

The implications of financial, strategic and compliance regarding to security, scalability, compliance and cost of local, remote or distributed solutions. (K13)

The uses of on-demand Cloud computing platform(s) in a public or private environment such as Amazon AWS, Google Cloud, Hadoop, IBM Cloud, Salesforce and Microsoft Azure. (K14)

Collate, evaluate and refine user requirements to design the data product. (S1)

Collate, evaluate and refine business requirements including cost, resourcing, and accessibility to design the data product. (S2)

Design a data product to serve multiple needs and with scalability, efficiency, and security in mind. (S3)

Automate data pipelines such as batch, real-time, on demand and other processes using programming languages and data integration platforms with graphical user interfaces. (S4)

Produce and maintain technical documentation explaining the data product, that meets organisational, technical and non-technical user requirements, retaining critical information. (S5)

Selects and apply sustainable solutions to contribute to net zero and environmental strategies across the various stages of product and service delivery. (S27)

Acts proactively and takes accountability adapting positively to changing work priorities, ensuring deadlines are met. (B1)

Data product deployment and evaluation
K2 K4 K8 K15 K17 K19 K20 K24 K25 K26
S6 S9 S16 S24 S26

Methodologies for moving data from one system to another for storage and further handling. (K2)

Frameworks for data quality, covering dimensions such as accuracy, completeness, consistency, timeliness, and accessibility. (K4)

Deployment approaches for new data pipelines and automated processes. (K8)

Data warehousing principles, including techniques such as star schemas, data lakes, and data marts. (K15)

Approaches to data integration and how combining disparate data sources delivers value to an organisation. (K17)

Differences between structured, semi-structured, and unstructured data. (K19)

Types and uses of data engineering tools and applications in own organisation. (K20)

Processes for evaluating prototypes and taking them to implementation within a production environment. (K24)

The lifecycle of implementing data solutions in a business, from scoping, though prototyping, development, production, and continuous improvement. (K25)

Data development frameworks and approved organisational architectures. (K26)

Systematically clean, validate, and describe data at all stages of extract, transform, load (ETL). (S6)

Query and manipulate data using tools and programming such as SQL and Python. Manage database access, and implement automated validation checks. (S9)

Develop algorithms and processes to extract structured data from unstructured sources. (S16)

Evaluate the strengths and weaknesses of prototype data products and how these integrate within an organisation’s overarching data infrastructure. (S24)

Identify data quality metrics and track them to ensure the quality, accuracy and reliability of the data product. (S26)

None

Collaborative working
K30
S22 S23
B2

The methods and techniques used to communicate messages to meet the needs of the audience. (K30)

Develop, maintain collaborative relationships using adaptive business methodology with stakeholders such as, business users, data scientists, data analysts and business intelligence teams. (S22)

Present, communicate, and disseminate messages about the data product, tailoring the message and medium to the needs of the audience. (S23)

Works collaboratively with stakeholders and colleagues, developing strong working relationships to achieve common goals. Support an inclusive culture and treat technical and non- technical colleagues and stakeholders with respect. (B2)

Professional discussion

KSBS GROUPED BY THEME Knowledge Skills Behaviour
Data quality and performance
K1 K3 K5 K18 K27
S7 S15

Processes to monitor and optimise the performance of the availability, management and performance of data product. (K1)

Data normalisation principles and the advantages they achieve in databases for data protection, redundancy, and inconsistent dependency. (K3)

The inherent risks of data such as incomplete data, ethical data sources and how to ensure data quality. (K5)

How to use streaming, batching and on-demand services to move data from one location to another. (K18)

The principles of descriptive, predictive and prescriptive analytics. (K27)

Work with different types of data stores, such as SQL, NoSQL, and distributed file system. (S7)

Optimise data ingestion processes by making use of appropriate data ingestion frameworks such as batch, streaming and on-demand. (S15)

None

Problem Solving
K21 K22 K23
S8 S10 S12 S18 S19 S20
B5

Policies and strategies to ensure business continuity for operations, particularly in relation to data provision. (K21)

Technology and service management best practice including configuration, change and incident management. (K22)

How to undertake analysis and root cause investigation. (K23)

Identify and troubleshoot issues with data processing pipelines. (S8)

Communicate downtime and issues with database access to stakeholders to mitigate the operational impact of unforeseen issues. (S10)

Maintain a working knowledge of data use cases within organisations. (S12)

Develop simple forecasts and monitoring tools to anticipate or respond immediately to outages and incidents. (S18)

Identify and escalate risks with suggested mitigation/resolutions as appropriate. (S19)

Investigate and respond to incidents, identifying the root cause and resolution with internal and external stakeholders. (S20)

Use initiative and innovation to problem solve and trouble shoot, providing creative solutions. (B5)

Regulatory Compliance
K10 K11
S13

Concepts of data governance, including regulatory requirements, data privacy, security, and quality control. Legislation and its application to the safe use of data. (K10)

Data and information security standards, ethical practices, policies and procedures relevant to data management activities such as data lineage and metadata management. (K11)

Use data systems securely to meet requirements and in line with organisational procedures and legislation. (S13)

None

Continuous Improvement
K16 K28
S11 S14 S17 S21 S25 S28
B3 B4

Principles of data, including open and public data, administrative data, and research data including the value of external data sources that can be used to enrich internal data. Examples of how business use direct data acquisition to support or augment business operations. (K16)

Continuous improvement including how to: capture good practice and lessons learned. (K28)

Evaluate opportunities to extract value from existing data products through further development, considering costs, environmental impact and potential operational benefits. (S11)

Identify new tools and technologies and recommend potential opportunities for use in own department or organisation. (S14)

Apply and advocate for software development best practice when working with other data professionals throughout the business. Contribute to standards and ways of working that support software development principles. (S17)

Identify and remediate technical debt, assess for updates and obsolescence as part of continuous improvement. (S21)

Assess and identify gaps in existing tools and technologies in respect of implementing changes required. (S25)

Horizon scanning to identify new technologies that offer increased performance of data products. (S28)

Quality focus that promotes continuous improvement utilising peer review techniques, innovation and creativity to the data system development process to improve processes and address business challenges. (B3)

Takes personal responsibility towards net zero and prioritises environmental sustainability outcomes in how they carry out the duties of their role. (B4)

Continuous professional development
K29
S29
B6

Strategies for keeping up to date with new ways of working and technological developments in data science, data engineering and AI. (K29)

Implement personal strategies to keep up to date with new technology and ways of working. (S29)

Keeps abreast of developments in emerging, contemporary and advanced technologies to optimise sustainable data products and services. (B6)

Employers involved in creating the standard: BBC, Ministry of Defence (MOD), Corndel, EasyJet, Aviva, Compare the Market (BGL Group), British Airways Ltd, Birmingham City University, British Airways, Sainsbury’s Supermarkets Ltd, Ministry of Justice

Version log

Version Change detail Earliest start date Latest start date Latest end date
1.0 Approved for delivery 11/12/2023 Not set Not set

Crown copyright © 2024. You may re-use this information (not including logos) free of charge in any format or medium, under the terms of the Open Government Licence. Visit www.nationalarchives.gov.uk/doc/open-government-licence

Is this webpage useful?

Thank you for your feedback

Tell us about your experience