Screw DevOps, Let's Talk DataOps

Editor's Notes

1. The concept was first coined just a few years ago by a Senior VP Platform Engineer, Dave McCrory. It was an open discussion aimed at understanding how data impacted the way technology changed when connected with network, software and compute. He discusses the basic understanding that there’s a limit in “the speed with which information can get from memory (where data is stored) to computing (where data is acted upon) is the limiting factor in computing speed.” called the Von Newmann Bottleneck. These are essential concepts that I believe all DBAs and Developers should understand, as data gravity impacts all of us. Its the reason for many enhancements to database, network and compute power. Its the reason optimization specialists are in such demand. Other roles such as backup, monitoring and error handling can be automated, but the more that we drive logic into programs, nothing is as good as true skill in optimization when it comes to eliminating much of data gravity issues. Less data, less weight- it’s as simple as that.
2. Data gravity is the ability of bodies of data to attract applications, services and other data. ... IT expert Dave McRory coined the term data gravity as an analogy to the way that, in accordance with the physical laws of gravity, objects with more mass attract those with less.
3. There are larger data sources every day. Databases are at the center of this friction and the natural life of a database is growth. By 2020, a third of all data will be on the cloud and 58% of data will be comprised in big data. In the last two years, we’ve created more data has been created in the past two years than in the entire previous history of the human race. Data isn’t going to slow down, either. By the year 2020, about 1.7 megabytes of new information will be created every second for every human being on the planet. By 2020, we’ll grow from today’s 4.4 zettabyets to an approximate, but staggering 44 zettabytes, or 44 trillion gigabytes. And by 2020, a third of that data will pass through the cloud.
4. Per Forbes, by the year 2020, about 1.7 megabytes of new information will be created every second for every human being on the planet. more data has been created in the past two years than in the entire previous history of the human race. That data has to be stored somewhere and there’s a large chance it’s going to be in a relational data store.
5. Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure” The term DevOps was popularized through a series of "devopsdays" starting in 2009 in Belgium I made an attempt to introduce it to my local user group in 2012 and it failed miserably. Last year, made a second attempt to great success.
6. ■ DevOps adopters are likely to invest in people and tools over the next year as part of their implementation of the strategy: The top investment item is hiring new resources with necessary skills (63 percent global and 72 percent in the United States), followed by engaging a consulting firm (51 percent global and 53 percent in the United States). More training for dev and ops personnel was cited by 46 percent. ■ A need to improve quality and performance of the applications as well as the end customer experience are major drivers (44 percent and 42 percent in the United States, respectively). ■ Nearly all organizations with greater than average profit growth have experienced tangible benefits by adopting a DevOps strategy. About 95 to 97 percent of these companies have seen increased frequency of deployments of software and services as well as applications made available across more platforms. ■ Security and organizational complexity remain as major obstacles in adopting the strategy (28 percent and 27 percent respectively).
7. Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure” The term DevOps was popularized through a series of "devopsdays" starting in 2009 in Belgium I made an attempt to introduce it to my local user group in 2012 and it failed miserably. Last year, made a second attempt to great success.
8. George has worked with the team to pick out the right tools tools for their environment, including Git for their repository, Jenkins for collaboration and deployment, Juju for Security and they’re even using Ansible for some of the automation. They’re well on their way with a great introduction of tools.
9. Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure” George builds out the first scrum sprint and readies the team.
10. The team builds out the sprint backlog and plans out what they will do and have to accomplish in the two weeks they have for the sprint. Everyone is assigned their tasks and ready to begin.
11. Each morning starts out great, as they have their daily scrum standups, taking just a few minutes to get everyone on board, tasks assigned and goals on what will be accomplished for the day off the burndown list.
12. The developers are starting to trip over each other in their traditional waterfall data environment. DBAs are busy and having difficulty providing them the data they need. They all want to succeed, but without the data, they start to miss the deadlines for the daily scrum burndown list.
13. Our Developer, Dave, built out a new test script that has been automated, but needs to be tested against fresh copy of production to development and test. The problem is, the DBAs can’t get him the data fast enough through traditional methods, even when they use DevOps methodologies, It can’t fix the current technologies the DBAs are employing.
14. Over 80% of time is waiting for RDBMS, (relational databases) to be refreshed. Developers and Testers are waiting for data to do their primary functions. This allows for faster and less costly migrations to the cloud, too.
15. The developers are feeling under more pressure as they can’t get the data they need and the DBAs are pressured to get space, time and resources.
16. This has become an oxy-moron. Databases are becoming larger and DBAs are slowing down the process, so we’ll remove those that understand how to manage it best?
17. George is at his wits end trying to figure out how they will ever succeed at DevOps and at Scrum if they can’t get through a simple two week sprint.
18. Where we had that one department with money to throw at an odd project, buying a server, developing what they needed and then it was our problem, now we have companies doing 30% or more business auditing cloud projects to deem if they are viable or not. BUT WHAT HAPPENS WHEN THEY DON’T or we don’t address the problem??
19. Where we before had to worry about someone, some department purchasing a server for the developer to do what they need to build out a host with an app, database, etc. that would become mission critical, the cloud has evolved this into an art. Just create a cloud account, choose your database, create your app and it then becomes production. Arrow Electronics, a major US company that was a cloud reseller now does 30% of its business auditing cloud deployed applications/databases to report on lack of best practices, security issues, etc.
20. For a typical Fortune 1000 company, just a 10% increase in data accessibility will result in more than $65 million additional net income. Leveraging data coupld increase revenue by as much as 60%
21. Marc gets it. He sees how much he and his team is in demand and knows that something needs to change.
22. In computing, virtualization means to create a virtual version of a device or resource, such as a server, storage device, network or even a database. The framework divides the resource into one or more execution environments. For data, this can result in a golden copy or source that is used for a centralized location and removal of duplicated data. For read and writes, having unique data for that given copy, while duplicates are kept to singular.
23. RMAN duplicates, cold backup to restores, datapump and other archaic data transfer processes are time consuming.
24. By virtualizing, we remove the “weight” of the data. We know that 80% of the data won’t change between copies, so why do we need individual copies of it. Our source is then deduped and compressed to conserve more space.
25. I work with Delphix, so you would think I know our virtualization the best, but the truth is, I also know many other virtualization tools at a very detailed level. The amount of information I know on Oracle virtualization tools is pretty insane, in fact.
26. How do we “rewind” data and code changes now? Why should the DBA rewind changes made in dev and test? Why should you be the one to do this in test? Virtualization removes this. The Virtual databases are read and write, so even maintenance tasks, like DBCC’s can be offloaded to one. Ability to version control, not just the meta data, but the user data!
27. Point out the engine and size after we’ve compressed and de-duplicated. Note that each of the VDBs will take approximately 5-10G vs. 1TB to offer a FULL read/write copy of the production system It will do so in just a matter of minutes. That this can also be done for the application tier!
28. Each Virtual Database, (VDB) will no longer require space, (only background and foreground memory for SGA/PGA, etc.) and local redo logs. This is a considerable savings, but… If we take this a step further by embracing write changes only on blocks changed from the source, then we’ll experience 10-20 copies of a database in about the same space that one database requires.
29. Each Virtual Database, (VDB) will no longer require space, (only background and foreground memory for SGA/PGA, etc.) and local redo logs. This is a considerable savings, but… If we take this a step further by embracing write changes only on blocks changed from the source, then we’ll experience 10-20 copies of a database in about the same space that one database requires.
30. Each Virtual Database, (VDB) will no longer require space, (only background and foreground memory for SGA/PGA, etc.) and local redo logs. This is a considerable savings, but… If we take this a step further by embracing write changes only on blocks changed from the source, then we’ll experience 10-20 copies of a database in about the same space that one database requires.
31. We’re using Groovy for this to build in Delphix and have also added the plugin to make it easier for those DBAs that may not be as savvy with DevOps tools.
32. Package software into standardized units for development, shipment and deployment. A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.
33. The next step is moving to data pods. Containers are a buzz area of technology right now. If we’re talking Docker or Kubernetes, we know this is the way of the future. Instead of having locked, unique environments, the ability to package them as one, in a lighter and more flexible unit makes incredible sense. As a DBA, I rarely, if ever, just released code to the database. It was commonly to the database, the application and linked products. The ability to package and manage as a Data Pod is an impressive enhancement to the Developer, tester and DBA.
34. The next step is the ability to migrate to the cloud or from one cloud to another. Right now, 60% of customers are using 2-5 clouds on average. The ability to move a Data Pod from one cloud to another is incredibly powerful. Companies are spending increased time now just migrating to the cloud, but to other clouds and if it would be as simple as migrating a Data pod with a few changes to the new storage location, (i.e. cloud) that could save companies millions of dollars.
35. A data pod is a set of virtual data environments and controls built then delivered to users for self-service data consumption. It allows for self-management without the need for DBAs to manage standard processing, automate rebuilds and even remove need for backout scripts when development, testing and promotion goes wrong.
36. We refer to a container as a template in our product. Note that a data pod can be moved here or to the cloud…
37. Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure” The term DevOps was popularized through a series of "devopsdays" starting in 2009 in Belgium I made an attempt to introduce it to my local user group in 2012 and it failed miserably. Last year, made a second attempt to great success.
38. This may appear to be a traffic disaster of changes, but for developers with Agile experience, a “sprint” looks just like this. You have different sprints that are quick runs and merges where developers are working separately on code that must merge successfully at the correct intersection and be deployed. Versioning with source control is displayed at the top, using Virtual images. You can see each iteration of the sprints. In the middle section is the branches of that occur during the development process. A virtual can be spun from a virtual, which means that it’s easier for developers to work from the work another developer has produced. Stopping points and release via a clone is simply minutes vs. hours or days.
39. This is a cornerstone to developers and testers, so as DBAs, we know the pain when a developer comes to us to flashback a database and before that, recover or logically recover, (import or datapump) independent objects. What is The developer/tester could do this for themselves?
40. This is the interface for Developers and testers- they can bookmark before important tasks or rewind to any point in the process. They can bookmark and branch for full development/testing needs.
41. Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure” The term DevOps was popularized through a series of "devopsdays" starting in 2009 in Belgium I made an attempt to introduce it to my local user group in 2012 and it failed miserably. Last year, made a second attempt to great success.
42. Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure” The term DevOps was popularized through a series of "devopsdays" starting in 2009 in Belgium I made an attempt to introduce it to my local user group in 2012 and it failed miserably. Last year, made a second attempt to great success.
43. As Agile matured, there was a significant missing component on the operations side. DBAs were already inundated with gate-keeping impacts to production. Operations from the business side, began to see the risk of outages and loss in revenue and began to buy into DevOps practies. There is a clear scoring card method to DevOps. It only works if there is a complete circle from development through release and that means successful release increases, along with effiencies.
44. And yet we state that we won’t need DBAs? That data isn’t the center of challenge?
45. The business is able to provision new environments or refresh existing ones in a matter of minutes. Developers and testers who’ve worked with bookmarks and branching of their code changes can now do the same with database changes, rewinding and refreshing as they need without impacting the DBAs day. This allows the DBA to do more with their time. Having tools that includes the database in the Agile development cycle makes a pivotal change in how the DBA is capable of being part of DevOps.
46. Two books at the bottom…they’re homework