Despite all the hard work of the last 25+ years, we still face enormous challenges in Oil & Gas data management. Recently, the focus has shifted from managing structured data repositories to the enormous volumes of ‘dark data’ that most of our clients still have in their digital vaults.
Gareth Smith, Head of Consulting at Sword Venture, explains that this shift is driven by the need to feed large volumes of high-quality data into analytics and data science-driven processes; growing regulatory pressure to report data; and the need to ensure safe, efficient operations (especially for the many assets that have swapped hands). However, most of this data is locked away in collections of documents and legacy proprietary formats, is nearly always poorly indexed and is often not machine readable. We have seen a surge in requests to help our clients do something about this.
Most of these Oil & Gas data management projects break down in to three core challenges:
Crucially, this has to be done without years of manual effort. It needs a different approach, using highly automated data science and analytics solutions to tackle these core challenges at scale and pace.
The first step is to build a pipeline that can ingest, store, process and extract/index data at scale. Unless our clients have the required compute resources to hand, we make use of the major public cloud platforms, e.g. AWS (Amazon Web Services) and Azure (by Microsoft), to provide the tools and processing power to tackle this challenge. A re-usable, cost-optimised and efficient data processing architecture based on cloud infrastructure reduces the costs and overhead to the client and allows us to move quickly through this stage of the process.
The next step is to clean up the data, understand what we have and identify the data that has some value to us. This is where we apply data science and analytics to automate and massively speed up what once was a laborious, labour intensive process. For example, we have developed a model using a machine learning algorithm, or neural network to predict the classification of a given document based on its text content, contained images and structure. Accuracy levels are often in excess of 95%. We combine this with analytics-driven clean-up and cloud-based optical character recognition (OCR) to create a repository of well structured, machine readable content ready for further analysis and processing.
The ‘Find’ and ‘Sort’ stages are just a means to an end; the goal is to extract value from data by putting it to work. We use a combination of machine and deep learning to identify and classify specific data within a document or text and extract that data in a machine readable format. For example, identifying deviation survey data (essential to all well interpretation) within documents and scanned images, extracting and making available to engineers.
We are working on the use of natural language processing (NLP) and machine learning techniques to draw meaning out of documents in order to generate greater insight. For example, to automatically recognise the findings and outcomes of final reports for thousands of wells, reducing the requirement for manual analysis.
Our goal going forward is to develop and deploy techniques such as deep learning and data engineering to synthesise large amounts of information and provide recommendations to assist human decision makers. For example, assisting with decisions on where to target exploration investment or how best to configure engineering parameters to reduce failure rates and maximise uptime. We recognise the value of combining data and knowledge to create unbiased predictive reasoning tools to support complex decision scenarios.
The combination of on-demand cloud computing and advanced data science and analytics techniques is revolutionising the way we manage data and extract value. We can tackle data at scale and pace, reduce manual effort and automate processes in a way that just wasn’t possible even a few years ago.
To find out more about how we can help you with your data management challenges, read on about our data and information management solutions here.
Gareth Smith, Head of Consulting at Sword Venture, delves into the detail of data management as an agile approach versus the more traditional waterfall project delivery commonly adopted in the UK Oil & Gas industry.
I sat in a fascinating session run by the Society of Petroleum Data Managers (SPDM), where Dan Brown, Executive Director at Common Data Access (CDA), talked about the challenges we face digitalising the Oil & Gas industry in the UK. Despite the many billions of IT spend over the years, apparently we still sit at the bottom of most ‘digital maturity’ league tables. One of the key questions raised was are we ‘agile’ enough in the way we work to reach the goal of delivering a digitalised basin on the UKCS? Does the long-term, physical asset management mindset of our industry, more aligned to manufacturing, lend itself to an agile approach?
I’ve discussed in the past about the rapid changes we’re seeing in Oil & Gas data management (DM); in particular the ability to apply new data analytics and data science techniques to address data management challenges at scale and speed. These approaches are more akin to science-based R&D or software development than the people-heavy, process-driven approaches of old. We are constantly finding and applying new tools and techniques to improve delivery and derive more value. The toolset is literally changing day-by-day as the underlying toolsets and algorithms improve and expand. The good news is this lends itself perfectly to an agile approach. In fact, it’s probably the only way you can do it.
Nearly all our clients are talking about agile. Some have already reengineered their internal IT and DM project delivery processes to take advantage of an agile approach, where the focus on an early realisation of value, continuous delivery and a mindset that accepts and embraces change. However, this is a massive shift in mindset from the days of 18-month+ project and budgeting lifecycles and waterfall project management that the industry has worked with for decades.
Many others say they want agile and all that it might deliver but can’t get over the line when it comes to the detail. An agile mindset requires an acceptance that the outcome is unknown. You can’t define the absolute time, effort and cost for an agile project. You can’t define a GANTT view of the project before you start. You work one sprint at a time towards your goal, focusing on continual value delivery and learning and adapting as you go. It also requires the continual involvement of the client, with day-to-day feedback guiding the work. As the client, you must retain ownership of the solution, not farm it out and hope that in 6 months’ time you have something of value back.
You try and narrow down the ‘cone of uncertainty’ as you progress through an agile project. You are always delivering value but there is always the chance you won’t do everything you set out to at the start, unless to decide to extend the project. You might argue that’s also the case with traditional project approaches, but it is baked in to agile. Change and uncertainty are core elements of the approach.
When it comes down to it, many say they want agile but really still want the (supposed) certainty of a waterfall approach to a project: a detailed scope & spec, a nice GANTT chart with lots of defined milestones and fixed estimates of cost, time and final deliverables.
Helping the UK Oil & Gas Industry deliver a digitised basin is going to take a conscious leap of faith to trust the relevance of an Agile approach to addressing the challenges in our industry. Read more about Sword’s data and information management solutions here.
Location intelligence is a key compnent of the solutions delivered to Sword clients. GIS Specialist Iwona Jakubowska shares her technical insights from the annual conference by Esri, an international supplier of geographic information system (GIS) software.
This year’s Esri User Conference was unlike any other. Billed as an immersive, fully virtual event, it gathered thousands of GIS users from all over the world. Over 80,000 users from 180 countries registered for the event that took place between 13th and 16th July 2020.
The event was designed to enable users to actively participate in the live sessions via chats, as well as watch pre-recorded sessions on demand. Esri has since made all of the sessions available online.
There were also some challenges, mainly around search/filter functionality (finding a proper session) and obviously time difference, as the event was hosted in Redlands, CA and watched live all around the world.
ESRI’s president Jack Dangermond led the plenary session, highlighting great Esri’s achievements in the field of Covid 19 response as well as many useful GIS implementations and solutions for various industries, like GIS integration with enterprise systems, including Microsoft Office, ETL (extract, transform and load) workflows, CAD (computer-aided design) and BIM (building information modelling). He announced a new education programme partnering with the National Geographic Society. Esri continues working with the UN on their Sustainable Development Goals.
Jack also announced new features and improvement to the Esri platform. He put a lot of emphasis on deep learning and AI (artificial intelligence) techniques available in the Esri platform.
The plenary session is available on the Esri Conference website.
The following developments in particular caught my attention:
ArcGIS Experience Builder is an application that allows users to create multi-page web applications using both 2D and 3D data. A great selection of tools and widgets enhances the look and feel, so that applications can be customised to a higher level. Users can now create new multipage, complex sites, which wasn’t possible before. Experience Builder is now available in ArcGIS Online (AGOL) and is soon coming to ArcGIS Enterprise.
Esri Notebooks are now available across the platform, from AGOL, through Enterprise to Desktop. There has been a Notebooks manager and admin API (application planning interface) added recently, that enables to create a custom template for your Notebooks. Multiple charts options are now available as well as a scheduler for automated tasks.
Python API enables administrators, publishers, analysts and data scientists to script and automate their workflows. All Python functionality is available through ArcGIS Notebooks. The ArcGIS Pro Python environment can also be extended with open-source libraries.
ArcGIS Analytics for IoT (internet of things) is a new real-time and big data processing and analysis capability in ArcGIS Online platform (an extension). It enables users to ingest, store, visualise and analyse data from IoT sensors. It allows to connect to almost any type of streaming data, perform real-time analytics and processing on the streaming data, and automatically disseminate information. It also enables users to design analytic models to process high-volume historical data in order to gain insights into patterns, trends, and anomalies in the big data.
In Esri terms: ArcGIS Analytics for IoT = (GeoEvent + GeoAnalytics) in the cloud
Voxel layer is a new data type introduced in the latest ArcGIS Pro release (2.5). Voxels are 3-dimentional cubes that can store various information. They allow you to visualise and explore multidimensional phenomena. You can now can easily create a cross section and slice through the voxel volume to investigate the distribution of the values in your data. Using transparency, you can focus on the most important information. You can also add fourth dimension, time, for better understanding the given phenomena (e.g. temperature). This video from Esri gives a great overview.
The conference brought together a host of insights into developments of Esri’s products and solutions and we’re really looking forward to bringing the benefits of these developments to Sword clients.
For more information about our information and data management please read more here.
Following on from Sword’s deep-dive into unlocking digitalisation value via our top 10 information and data management themes in Oil and Gas, Neale Stidolph, Head of Information Management & Energy Transition Lead, revisits these themes to consider the implications of Energy Transition.
Energy Transition is picking up speed and there are major investments being made, in a new landscape of operating companies, suppliers, regulators and many intersections and crossovers with the current Oil & Gas sector. There is an opportunity to take advantage of information and data for better business performance, and avoid delays, cost overruns, regulatory impact and to make best use of existing skills.
Digitalisation needs data, needs data management
There are new streams of data from sensors and applications, with a need to closely monitor greenhouse gas emissions. New collaborations are emerging, for example the Seagreen windfarm project. Connecting companies from different sectors is greatly helped by making information and data readily available and in a form that all parties can trust and know how to use.
Asset transfers and the crucial role of data
Just as decommissioning is a crucial factor when taking on an existing production asset, so is the status and trending of emissions. Plans can then be made for Energy Transition such as electrification through floating wind, wave systems or connecting into emerging renewable energy hubs. Good data will help make the case to the regulator when taking on additional assets.
Governance is making a comeback
Emissions goals are being set at increasing levels of detail, and this needs to be easily rolled up to see company performance all the way up to national emissions levels. Legislation is defining the macro targets and changes to carbon taxation are likely, as the UK is below other levels such as CO2 tax in Norway. Good data on CO2 output and trends at a granular level will help plan what can be captured and stored. The Oil & Gas Climate Initiative has a new CO2 storage resource catalogue that shows locations and potential capacity.
Stuck at the ‘proof of concept’ (PoC) chasm
As Energy Transition is in flux people are working on many aspects, however joining that up more collaboratively across the industry would help save time and money. Work on emissions dashboards and reporting is something that suits a group initiative. Currently offshore wind seems to be very much about individual projects, all tied back to shore. The emerging strategy is about a grander plan that would form regional offshore hubs and international connectors, as a backbone for enabling many more projects to move ahead. It seems similar in terms of information and data, as a clearer framework that helps everyone from suppliers to regulators makes life simpler. This approach has been applied successfully by the Oil & Gas Authority (OGA) and the National Data Repository (NDR), the new version of which is likely to include Energy Transition considerations next year.
Dropping down the digitalisation hype curve
Yes, there’s still lots of digitalisation hype around. The Energy Transition twist is that folk seem to be more open as it is new and less familiar. The questions are familiar enough. Companies need to know where they stand, see how they’re trending, make plans and monitor their effectiveness – which are all information and data driven. Hype is reduced if there are good sets of requirements emerging which people can get involved with and adopt. That’s how the Oil & Gas UK IM Forum is proceeding, to assess the current state, gather views and see what the OGTC, OGA and trade organisations in renewables are doing. There are dozens of trade groups which makes the landscape complex, hence best to navigate that intelligently via a group like the IM Forum. That will help avoid duplicated effort or ideological conflict. It looks like this work will link up well with the Open Subsurface Data Universe (OSDU) and their emerging New Energy charter.
Oil & Gas core business: hydrocarbons or tech?
In our last article we spoke about core Oil & Gas activities such as exploration and production and how that can be at odds with an entrepreneurial start-up IT approach to digitalisation. Those forces are still very much in play, but Energy Transition brings new angles to consider. Operators may choose to buy-in renewable power for offshore facility electrification, but some may acquire more capabilities of their own. How hands-on Operators decide to be is going to be interesting, to see how they will adapt.
Net Zero sentiment and “oil shaming”
The last few months have brought multiple impacts of the pandemic, oil demand collapse and resulting fall in Oil & Gas prices. Many conference calls and webinars have looked at the likely impact, will this hit the Energy Transition pause button? Will investors rush away from Oil & Gas? It seems not. That’s not to say it will be easy or that there will be winners and losers. The industry is necessary and contributes to cleaner energy through gas and hydrogen, and can capture and store huge amounts of CO2. This helps tremendously, accelerating plans by making best use of existing facilities. The new skills needed for Energy Transition could offset many potential job losses. This creates a more positive feel that will actually get things done.
The search for talent
Some skills will adapt, such as subsurface specialists working on reservoir suitability and long-term management for CO2 storage. New types of equipment will need to be designed, deployed, monitored and maintained, making best use of existing skills in those areas. We recently presented to the Society of Petroleum Data Managers on this topic, to help raise awareness so that people can consider how they can enhance their skills.
Collaboration is key
Having mentioned the OGA’s NDR it is also worth adding the changes going on in the ΩSDU, and their ‘New Energy Charter’ under development. Ensuring that these, along with data transfers standards all fit nicely together would be great. Best not to rush forward without taking a moment to see what is out there and how it is changing.
Many companies are involved in amazing projects such as Acorn, Northern Lights, and PosHYdon. It’s an exciting time to see the practical progress towards clean energy, see how our working lives are changing too, reducing dependence on travelling for work. We’re fortunate that smarter ways to work with information and data were put in place so that many people are now not dependent on being in offices. This digital revolution will play a key part in bringing about Energy Transition.
We turn the Sword Spotlight on Sam Omidi, who recently joined the Energy team as a Data Scientist. Here he talks us through a little background and reflections on his first few months with Sword.
“I think the most surprising thing I’ve learned is that I’m not a fake data scientist among real ones. They call them unicorns for a reason. I can now say with confidence that I can code and I do know software development lifecycle and statistics. Without these skills, life as a data scientist would be difficult and one would struggle a lot. In general, people have their strengths and experience in one or two things and are weaker at others. I now believe a data scientist needs to know more than just a couple of things. They need to know how to handle out-of-memory errors, data structures, and expectation maximization or where heteroscedasticity is coming from.
“I switched into a focus on data science and machine learning during my final year project in my bachelor degree by working on a data mining project and social network analysis. This interest truly materialised when I got Masters in Data Science.
“I joined Sword three months ago as I found the team very passionate about their work. Also, the projects they are working on are very unique and exciting.
“I got heavily involved in my first week with designing a data pipeline and evaluating various novelty detection algorithm, along with my lead. It has been absolutely fantastic seeing myself hitting the ground running from the start and feeling part of the team immediately.
“Additionally, in the subsequent projects I had a chance to boost my knowledge about neural networks and focus my research on them, developing more in-depth understanding.
“A typical day for me starts around 8:00am, when I catch up on my social media accounts related to machine learning and data science. I switch into work projects around 8:30am and finish around 5:00pm with a break for lunch. About 40% of my time is spent on research and development, with a strong focus in mathematics to be able to understand what is actually happening under the hood of every algorithm that might be a good candidate for our project. The work involves anything from developing and testing new algorithms to writing mathematical proofs to simplify data problems.
“Another 30% of my time is spent on research about best coding practice in data science, which often identifies problems related to data capture, data pipelines or specific implementation of an algorithm with needed modifications. This is probably one of the most crucial aspects of the job. As a data scientist, it is important to communicate with a wide range of stakeholders, so it helps simplify my explanations of machine learning algorithms to a layman's level.
“The other 20% of my time is meeting with my colleagues. We discuss the projects and support each other on our tasks.
“Each project is unique, and I try to let the project and its initial findings guide me to next steps. I mainly use Python and Azure platforms for projects, though R, and SAS are occasionally helpful with specific packages or R&D requests. I can usually recycle the code, but each problem has its own assumptions and data limitations with respect to the mathematics.
“Overall, it has been a great start and I am looking forward to what the future will hold for me and the team.
Sword Venture and Target Energy Solutions have signed a partnership agreement to deliver their technology and services as a combined offer in the areas of data foundations, data management and business consultancy on digital transformation solutions.
The two companies will combine their market-leading services and technology to deliver next generation trusted data platforms to the Oil and Gas industry that offers data integration and insights, reducing costs and value through better informed decision for its clients. This offer will include leading-edge loading automation, Google-like search and data science techniques to reduce manual overhead from a client and service delivery perspective and significantly improve data quality, accessibility, performance and regulatory compliance.
Gareth Smith, Head of Consulting with Sword Venture said; “We’ve been working with Target for some time to explore opportunities to leverage and integrate our mutual capabilities. We are both developing new technologies and approaches to tackling the data and information challenge all our customers face on a day-to-day basis. We feel the combination of Target’s MEERA Trusted-Data platform and our data science and analytics-led services provide a potential step-change. There’s never been a better time to unlock the value of data in Oil and Gas and address the very real challenges we face around supporting energy transition, improving operational efficiency and managing risk.”
Matthias Hartung, President Digital Transformation with Target Energy Solutions added; “More than ever, our industry is demanding data foundation and management capability to enable cost control, adoption of digital technologies in pace and scale, and insights to address current economic, environmental and social-acceptance challenges. Finding and getting trusted data to any end-user app requires data foundation technology and domain-competent services which is what the partnership of Sword and Target can jointly bring. We look forward to working our new partnership with Sword to meet market demand.”
Target is an international technology and services company focused on delivering on enabling digital transformation solutions for data-driven industries . Formed in 2004 by Oil and Gas industry professionals and headquartered in Muscat, Target now operates globally with 9 offices from Houston to Beijing and a total staff of 750. Target has expanded its client list to include regulators, NOCs, IOCs, investors and service companies around the world.
For more information visit https://www.target-energysolutions.com/
Formed in 1990, Sword Venture designs, implements and supports practical technology, data and information solutions that enable Oil and Gas companies to gain insight and make effective business decisions. Sword Venture is part of the global Sword Group, an international consulting, services and software company and global leaders in digital and technology transformation, with over 2,000 employees working in 50 global locations.
Read more here about how we help to design, implement and support practical information management services and solutions.
Following the recent acquisition by the Sword Group of DataCo, Sword Venture and Sword DataCo will be merging to form a single entity that will specialise in Energy sector data services and solutions. The combined organisation will be known as Sword Venture from April 1st 2020. This is a natural progression that builds on the excellent working relations we have fostered since joining forces last year.
With a combined workforce of over 120 data and information management professionals, Sword Venture brings unparalleled expertise in maximising the value of Oil and Gas data and information assets for our customers, via capabilities such as:
Sword Venture and DataCo have both invested in data science and analytical capabilities in recent years, bringing these two teams together as the Sword Venture Digital Solutions Factory will provide our customers with class-leading technology. The Factory comprises of an in-house R&D department and a commercial analytics delivery team, with the goal of solving common Oil and Gas industry problems across the technical and corporate domains. Our blend of technical data scientists and data domain subject matter experts form agile delivery groups that deliver practical value to our customers.
We focus on delivering solutions that enable insights and inform business decisions, through the application of data science and analytical techniques. Our solutions are proven to help reduce time, cost and risk, and improve insight and efficiency in operational decisions. For example, in a complex data migration triggered by organisational change such as an asset transfer, we bring an independent lens to the discovery, access and use of data. We ensure the vendor extracts only the information relevant to that asset and that the new owner is armed with the data, information and licenses needed to deliver seamless operational production from Day One of ownership.
As part of Sword Group, we also have access to IT, visualisation, and digital workplace expertise to help build solid technology foundations that are designed to enable successful digital transformation and embed intelligent data solutions.
We have the joint ambition to firmly establish ourselves as the leading global provider of data and information management expertise and service support to the Energy sector. Building on our existing global footprint from Sword Venture (with offices in London and Aberdeen) and Sword DataCo (with offices in Aberdeen, The Hague and Perth, Australia), we are actively looking to expand our presence in locations that include Houston, the Middle East, and South East Asia.
We are confident that this integration will enhance the service we provide to our customers today, as well as help us to shape the next decade of data management in the Energy sector together.
Read more here about how we help to design, implement and support practical information management services and solutions that enable our Energy sector customers to gain insight and make effective business decisions.
We are looking for suitably qualified Document Controllers to work across our O&G client sites in Aberdeen.
We are currently interested in filling the following Document Control positions:
The following skills and experience must be demonstrated:
If you have the relevant experience and are interested in this position, please contact Malcolm Grant in the first instance.
Recent research between Sword IT Solutions and AIIM has shown that there are 6 key reasons why business are failing to deliver effective digital workplaces:
Download our info graphic for more details......
Implementing an entirely new, greenfield IT infrastructure in the place of your new acquisition’s existing, outdated system can seem like a daunting and complex process. First, there’s the never-ending timescale. There are the intricate project management challenges: inevitable when managing multiple parties and new working relationships. Then there are the technical challenges, which include design, implementation and the safe and secure migration of data from many and varying sources - and all whilst maintaining service availability.
Yet while the pressures may seem insurmountable, these are not reasons to stick with an old, outdated system. As high as the mountain may seem, in the hands of the right workforce, with the right knowledge and the right skills to deliver on time, on budget and the rest, it’s really only a molehill.
Edinburgh Airport (EAL) is one of the busiest, most bustling airports in the UK, with over nine million passengers passing through each year and over three hundred flights in operation every week. When GIP bought the airport from BAA in 2012, they had an almighty task on their hands: to put in place a whole new IT infrastructure as a replacement for the old, operational system. And all with the additional pressure of a particularly tight twelve month timescale.
EAL selected Charteris to design and implement a new SharePoint infrastructure and, following that, to migrate a wealth of applications and data running on the old BAA system: SharePoint 2007. EAL had a strong financial incentive to complete the program on time and even EAL’s Architecture and Investment Manager himself, Kevin Norman, felt the challenge was almost too much. ‘When I arrived,’ he admits, ‘I felt the timescales were too aggressive to complete the separation.’ Yet what others had labeled a three year project, Sword IT Solutions completed in just nine months.
Sword ITS chose to use SharePoint Server 2013 to build the new infrastructure, the initial requirements of which were light on detail and needed a significant amount of design. With a combination of migration and new development, the team transferred to it, with the help of the agentless MetaVis, a variety of applications and data that were previously running on the BAA SharePoint 2007, Lotus Notes and various MS Access systems.
The project was completed on time, on budget and with great success, in spite of the tough time pressures, a complicated programme environment and an unrefined set of requirements. Sword ITS’ agile and flexible approach, with sound decision-making and effective project management, meant that the team delivered without overspending and within nine months. Due to its flexibility, reliability and the fact that it can be easily accessed from anywhere in the world, the SharePoint platform is now the platform of choice for any new application functionality at EAL.
With a good team and using SharePoint technology it is possible to navigate the project management and IT development quagmire to rebuild your systems – and it doesn’t have to take three years, even on a project as complicated as one of the UKs biggest airports. Despite the pressures, it’s worth implementing new technology to replace old, outdated systems, no matter how complex the project may seem.