MySQL X DevAPI 8.0.3 for PHP is here!

Dear MySQL users,

I’m glad to announce that MySQL X DevAPI extension for PHP 8.0.3 has been recently released!

If for some reason you don’t know what this is all about, then in short the MySQL X DevAPI for PHP add support for the new X DevAPI in PHP, and it’s released as an extension to the language. The X DevAPI enables application developers to write code that combines the strengths of the relational and document models –MySQL as a Document Store– using a modern NoSQL-like syntax that does not assume previous experience writing traditional SQL.

For general documentation about how to get started using MySQL as a document store, see “Using MySQL as a Document Store”.

Download the extension.

Depending on your operating system and working environment, you can download the extensio from:

  • Direct link to the mysql_devapi extension pecl web page, here.
  • On Fedor-ish Linux system, use the Remi’s repo.
  • Clone the source code repo, here.

Further instruction on how to install the extension with samples are included in the readme script and documentation.

This new release contains many bug fixes, refactoring’s and improvements that are a step forward in our work to improve the quality of the code base and thus of the extension itself, beside the minor tasks we were able to deliver some big features:

Support for Array or Object “contains” operator:

New CONTAINS and NOT_CONTAINS operators introduced in any expression valid in CRUD operations. They are different from the IN operator, in that IN works with SQL and requires a set of literals at the right side. CONTAINS should accept JSON values at the left and right sides of the operator,including expressions that generate a JSON value.
Example (Where $coll is a Column object):

New API’s: getOne, removeOne, replaceOne, addOrReplaceOne:

This extension is specific to Collections. These are commands that operate at a single document level, unlike the other CRUD commands that operate on all documents that match a filter the following collection commands are introduced as a set of direct-execution complementary operations that reference documented by id, as opposed to a free-form expression.

When working with small documents (majority of the time), the simplest and easiest way to change a certain document is to use a load/modify/save pattern:

New row locking mechanism for Crud.Find / Table.Select

The MySQL SELECT statement supports now locking matching rows, for reads and for writes (SELECT … FOR UPDATE or SELECT … LOCK IN SHARE MODE).

The find() CRUD method is extended to support this feature, allowing safe, transactional document updates on collections. Transaction support is an important MySQL differentiator compared to other NoSQL databases. Example:

Client A execute:

Client B execute:

Then Client A commits:

And finally Client B can continue:

Support for the new MySQL Server 8.x, including fixes for phpize build.

Large refactorings and fixes where made by the team to allow this PHP extension to fully support he new MySQL server 8.x, also the problems with the phpize build are now solved and we’re happy to deliver a rock solid release with plenty of support for the new MySQL’s.

Thanks for reading,

On behalf of the MySQL/Oracle X DevAPI for PHP Team, Filip Janiszewski

Simplified Client and Connector Versioning

Here at MySQL we are constantly improving both our servers and our client software by making things simple to use and understand.  An area where we want to simplify things is around the version number we use for various MySQL client, connector, and other applications like Workbench, MySQL Enterprise Backup, etc.

Basically:

  • We want to make sure our users easily know which version of a connector or client works with which server version(s).
  • We want to deliver support for new server features during the server DMR phase without disrupting the GA versions of the connectors.

Which Connector Version To Use?

MySQL connectors (our client libraries — Connector/C, Connector/J, Connector/Net, etc) and other clients (tools and applications — MySQL Shell, MySQL Enterprise Backup, etc) will generally release more often than the server.  Relative to upgrading a set of servers in an enterprise, upgrading a client is normally quite simple.

Because of this rapid versioning of clients we have historically not tied the client version to the server version.  In addition to that we’ve changed the server versioning as of MySQL 8.0 to increment just the first digit for each major release.  After 8 comes 9, etc.

So you may see that the current versions of some client libraries are all over the place – 6.9 for Connector/Net, 5.1 for Connector/J, 2.1 for Connector/Python, or 4.1 for MySQL Enterprise Backup.  As you can see these version numbers do not relate to the server version at all.  We think this is confusing and with the recent server version change we could simplify things for clients, connectors, and applications.

Note:  In some cases we tried to make this easier by having a policy that all our support client libraries must support all servers that are still supported, but then there are other clients like MySQL Enterprise Manager where that’s not the case.

Challenges – DMRs and Testing Out New Server Features

The server team provides pre-release versions of the server available for testing out new features.  These releases are called DMR or “Development Milestone Release”.  While many of the new server features are transparent to client libraries, some require client library updates like new collations or new security measures.

When server changes affect clients users ask how can they try out the new features?   What connector or client version should they use?

And, features appearing in server DMR releases can change.  They are, after all, in development so adding support for pre-release features that can change into a GA connector is really a no-go.  So we need a  way to make new server features available to connectors and clients without disrupting the normal feature cycle of the clients.

Solving – New Simplified Versioning Methods for Clients

To address these issues beginning with the 8.0 server GA release, all of our connectors will synchronize the first digit of their version number with the server.   This means that MySQL Server 8.0 will go GA alongside 8.0 versions of each of our connectors.

Now, it’s easy to know which version of a client you should use – the first version number should match (starting with 8.x.x releases).

  • For Connectors – Using MySQL 8.0 then any of our MySQL connectors 8.0 or later will fit the bill!  Does this mean that older connectors will not work with MySQL 8.0?  Not necessarily!  However, older connectors might not be quite as compatible and will certainly not support many of the new features.
  • Other client tools and application – Will also go to 8.0 versions – MySQL Shell 8.0, MySQL Workbench 8.0, MySQL Enterprise Backup 8.0, etc.

As new MySQL Server DMR releases come out you can expect to see corresponding clients to help you try out the new features in both the client and the server.

We see the change as an opportunity to really clarify our product offerings using the version number and deliver a much more unified experience.  Clients will still version frequently so you’ll see some clients move ahead with 8.1 and 8.2 versions but that first digit 8 will still tell  you that it will work perfectly with MySQL Server 8.0

We’ll have more to say on this in the coming weeks!  Please let us know if you have any questions!

MySQL community recognition

On behalf of myself and the MySQL Engineering team, I’d like to thank everyone for naming Oracle the Corporate Contributor of the year at the MySQL Community awards ceremony at Percona Live 2014.

I’d also like to thank everyone who have come up to me and my fellow colleagues in person to thank us for the great work they have seen in MySQL 5.6 as well as 5.7. I only wish that more people in the MySQL Engineering team would be able to share that experience.

Therefore I cannot stress enough how important it is for my team to hear such positive feedback through talks, blogs, and tweets. Thank you for doing so, and please continue sharing your positive experience with MySQL openly. Why not read about the latest mysql engineering work and leave a comment on mysqlserverteam.com, it is a huge motivator.

Tomas

Announcing new Yum repositories for MySQL

The MySQL Engineering Team at Oracle is excited to announce availability of Yum repositories for MySQL, making new releases of MySQL Database and related products easily accessible using Yum.  This initial release is focused on EL6-based distros as well as Fedora 18 and 19, and provides easy access to the most recent GA releases of our most popular products:

  • MySQL Database 5.6
  • MySQL Workbench 6.0
  • MySQL Connector/ODBC 5.2

We expect to expand this in the future to offer additional MySQL products and versions using these repositories, as well as repositories for additional Linux distributions.

This effort benefits both end users and Linux distributions.  Users will have additional choice in deploying specific versions of MySQL products using their favored package manager, while Oracle helps reduce overhead in packaging MySQL for specific distributions.  For users, the benefits are:

  • Ability to get MySQL components not yet packaged by their Linux distro.
  • Access to the latest and greatest versions of MySQL, even if they have not yet been adopted by their Linux distro.
  • Packages of MySQL products supported by Oracle.
  • Easy early access to pre-GA product releases, such as MySQL 5.7 Development Milestone Releases.

Equally important in driving this effort is our continued and expanding commitment to Linux distributions.  For this audience, the key benefits are:

  • Increased attention to packaging needs and concerns at Oracle means fewer opportunities for changes to be introduced which cause downstream packaging problems without awareness first at Oracle.
  • Maintainers have access to our packages as a template for solutions to packaging issues.
  • Oracle will have increased, distribution-specific experience and knowledge to draw upon in support of maintainer requests.
  • Packaging work will start before product GA at Oracle, allowing packaging to mature concurrently with the product it supports.

We’re excited to get MySQL 5.6 – the best release of MySQL Database ever – into the hands of more users, and to invest more in supporting maintainers in their efforts to package MySQL products for use within their Linux distributions.  These packages have been extensively tested internally, fully supported, and install fully tested and supported MySQL products.  Listed below are several resources you may find useful in deploying MySQL using the Yum repositories:

Please let us know about your experiences with these new repositories, as well as any suggestions you may have to improve the experience.  We welcome feedback – both bug reports and feature requests – via bugs.mysql.com (there is a new “MySQL Repositories” category).

Also have a look at Norvald’s blog post documenting the team’s work producing these packages

Join us at MySQL Connect

I’m excited about this year’s MySQL Connect in San Francisco. It’s bigger than ever, with 3 days of sessions, tutorials, and hands on labs.  We have more MySQL engineers going there than ever before. Looking at the keynote we are preparing, we have a lot of great news, and exciting benchmark numbers to share.

If you have a chance, make sure you join us this year. Learn about the latest MySQL engineering development, and use this opportunity to talk to the MySQL engineers themselves.

Hope to see you there, Tomas

 

MySQL Server 5.6.13 Community Release Notes

MySQL Server 5.6.13 has been released, and is available (as always) in GPL-licensed Community builds as well as commercial-license builds for evaluation and customer use. By my count, the release notes show just over 100 bugs fixed, improving user experiences both for community and customer users of MySQL 5.6 alike. The MySQL community was an integral part of that effort, submitting almost 40 of the bug reports fixed in 5.6.13. I’m taking this opportunity to express my gratitude on behalf of the MySQL Engineering team at Oracle for these efforts.

  • Po-Chun Chang has identified a series of code optimizations, frequently providing patches. In Bug#69377, this comes in the form of an optimization by eliminating needless work in an internal InnoDB function.
  • Multi-threaded slaves could hang on binlog rotation, due the Bug#69369 identified by Santosh Praneeth Banda. We appreciate the patch, even though we ended up fixing it differently.
  • Bug#69341 from Yoshinori Matsunobu exposed a bottleneck with semi-sync replication with many clients, and while there technically wasn’t a patch submitted, Yoshinori’s deep code analysis pointed the way forward.
  • Yoshinori was similarly responsible for Bug#69316, which reduces DROP/ALTER TABLE time for InnoDB tables.
  • Davi Arnaut found an optimization opportunity in InnoDB’s handling of concurrency tickets during row reads in Bug#68869.
  • InnoDB’s new FULLTEXT now mirrors that of MyISAM with respect to leading wildcards in search terms, as a result of Zoltan Fedor’s report in Bug#68949.
  • Davi contributed again with his report of Bug#68501, which allowed InnoDB to keep under-filled index pages incorrectly.
  • InnoDB tables could be left in an inconsistent state during ALTER TABLE commands when foreign key checks are disabled, as reported by Ernie Souhrada in Bug#65701.
  • Lawrence Holtsclaw reported in Bug#61656 that CREATE TABLE statements with comments containing escaped apostrophes could trigger silently omit the foreign key definitions.
  • Thanks to Kolbe Kegel’s two related bug reports – Bug#68602 and Bug#68599 – warnings issued related to the storage of replication credentials are clarified.
  • Bug#68460 fixed a hang where FLUSH TABLES WITH READ LOCK was followed by replication updates and a subsequent SHOW SLAVE STATUS.
  • Calvin Sun identified a regression in Bug#69127 related to how InnoDB would trigger server-level handling of deadlock conditions in subqueries in an UPDATE statement.
  • Bug#59905 addresses plugin compatibility on certain platforms, such as PPC and ARM.
  • Alexey Kopytov found unoptimized macros being used on certain platforms in Bug#61179.
  • mysqldump will no longer fail when dumping from older MySQL Server instances which don’t have the mysql.general_log and mysql.slow_log tables, thanks to Bug#65670 reported by Van Stokes.
  • Chito Angeles found a problem with InnoDB’s handling of apostrophes in BOOLEAN FULLTEXT searches in Bug#69216.
  • Jan Rusch also contributed an InnoDB FULLTEXT search bug report in Bug#68720, which prevented modifiers from being used in conjunction with quoted literal phrases.
  • MySQL will now recognize those storage engines which declare that they use extended secondary keys for certain optimizations that were previously hard-coded for InnoDB, thanks to Bug#68469 reported by Artem Livshits.
  • Dan Kloke found in Bug#52582 that certain multi-table COUNT(DISTINCT) queries when big_tables were enabled returned wrong results.
  • Per Bug#68438, mysql_install_db would fail against MySQL instances built without InnoDB. Thanks to Rene Berber for the bug report!
  • CREATE TABLE IF NOT EXISTS no longer requires exclusive metadata locks as a result of Michael Skulsky’s Bug#63144.
  • Colin Charles and Hye Chung each identified potential RPM build problems from source due to a missing pb2user in Bug#63144 and Bug#69339, respectively.
  • Due to a typo in CMake scripts identified by Takashi Ichii in Bug#60743, DTrace wasn’t properly enabled.
  • Takashi Ichii also identified a performance issue with PERFORMANCE_SCHEMA in Bug#69382.
  • In Bug#68897, Helmut Lange found a defect which could return wrong results for certain LEFT JOIN queries with GROUP BY clauses.
  • Yoshinori identified a misleading error message related to global transaction IDs in Bug#69096.
  • Bug#62769 identified another portability problem and was solved.

Thanks to all of the above community contributors who invested time in reporting bugs, and especially to those who dug into code to provide analysis or patches.

Tomas

The MySQL Man Pages ARE Available under the GPL

Due to a bug in our release packaging scripts, the wrong version of the man pages, with the wrong license text, were copied into the MySQL Server GPL packages. The MySQL Man Pages continue to be available under GPL. The MySQL Server GPL packages will be corrected ASAP. You can read and follow the public bug here.

We apologize for the confusion this has caused. As always, please feel free to contact us, to ask about changes that you are wondering about.  Reporting a bug is always a good way to communicate with us.

Have a Happy GPL Midsummer, Tomas

April MySQL Engineering News

On a regular basis I plan to summarize the latest news from MySQL Engineering. I hope you find it useful.

April highlights were the DMR’s coming out for both MySQL Cluster 7.3.2 and MySQL Server 5.7.1. For those that have been missing the launchpad versions of those, I apologize for the delay, but they should be there now if you want to dig into the changeset details. And to repeat what I’ve said in the past, there should not be a delay between releases on launchpad and src tar balls, so please keep bugging me when you see glitches there. Personally I also very much enjoyed the April Percona conference where I met a lot of old good friends and gave a keynote. You can see it here or just read the presentation. Here follows a summary of the engineering blogs from last month.

MySQL Support
The MySQL support team has some excellent blogs well worth reading.  The latest month includes for example “Migrating to InnoDB Full text search…”, “MySQL Utilities…”, and plenty of recent 5.6 info.

MySQL Performance Schema (PS)
This is a tremendously powerful tool, but it can be overwhelming to familiarize yourself with. Learn more about configuring performance schema from the author Marc Alff. Feel free to give him feedback directly on his blog, the good as well as constructive feedback. I’d also like to draw your attention to an older blog by Mark Leith, ps_helper. We are currently considering how we could include something like that in the product to make is easier to use PS. Ideas and input are greatfully received.

MySQL InnoDB
InnoDB continues to be the storage engine we focus on together with NDB for MySQL Cluster. You can learn more about the Data Organization in InnoDB, and the new 5.6 feature InnoDB transportable tablespaces and how to use them. Also have a look at Sunny’s presentation about 5.6 InnoDB features and scalability at Percona Live in April

MySQL Cluster and NoSQL
I mentioned the new DMR 7.3.1 for MySQL Cluster. Our efforts to improve ease of use of Cluster continues, please learn more about, and try out, the autoinstaller. MySQL Cluster was there as a “NoSQL” database long before the term was coined, and was just awarded the “storage engine of the year” the other week. This DMR also contains support for Foreign Keys which I know some of you have been waiting for. I’ll encourage the cluster engineers to write an in depth blog about the complexities in supporting foreign keys in a distributed databases. Meanwhile please get an overview here.

MySQL Performance
Dimitri is continuing his great work on benchmarking MySQL 5.6, comparing it with different 5.5 versions, including MariaDB and Percona servers. His blogs are excellent reads for anyone interested in MySQL performance.

MySQL Testing
In my keynote at Percona Live I talked about the big investment we have made in QA for MySQL. Please take a moment to learn about MySQL Server QA.

MySQL Replication and Big Data
MySQL 5.6 contains many new replication features, learn about the lesser known but equally important new replication features. Luis gave an excellent presentation at Percona Live on MySQL Replication, where you can learn more about performance enhancing features like multi-threaded slave and binary log group commit, as well as global transaction identifiers that can help you to do failover much better.

In my keynote at Percona Live I talked about how MySQL plays in important part in the Big Data lifecycle, and the new Hadoop applier for MySQL. Here is some more reading from Shubhangi, Hadoop applier part 1 and part 2.

MySQL Optimizer
MySQL 5.6 contained a lot of Optimizer enhancements. Here is a blog on subqueries from Øystein, a senior member of the Optimizer team, also aggregated to the MySQL Optimizer blog if you are interested in older posts.

MySQL on Windows
Our focused effort to on MySQL on Windows continues. New this month is the beta of the Connector/Net 6.7.2 which includes support for connecting to the MySQL memcached interface as well as load balancing. If you have not yet tried out MySQL for Excel I encourage you to try the newest version. You can follow the development of MySQL on Windows blog if you are interested.

MySQL General
Please have a look at Mikael’s insights about MySQL Engineering. And read about the fixing of a long standing trigger bug which is also aggregated on the MySQL Runtime blog.

MySQL Cluster gets a Community Award

It was with pride I entered the stage at the Percona MySQL Conference yesterday to receive the “Storage Engine of the Year 2013 award” for ndb (the MySQL Cluster storage engine).  Thank you very much, on behalf of myself and the MySQL Cluster engineering team. The team has done a great work over the past 10 years to harden the product, and add new features. MySQL Cluster milestone 7.3.2 was just released. It will be my pleasure to present the “trophy” some members of the team when I get back to Stockholm. Please see a previous post for how the MySQL Cluster journey began for myself and the ndb team 10 years ago.

The milestone release model revisited

In my Percona live keynote “Driving MySQL Innovation” on Tuesday I talked about the quality problems we had with 5.0 and 5.1 and the switch to the new Milestone Release model. I explained how this new development process allowed us to deliver both 5.5 and 5.6, on time, including innovative features, and with very high quality.

The description of the Milestone Release model has been available here, for almost two years. So, this is not new but it has now been in operation for 3.5 years and it can be worth looking back and summarize.

Our first MySQL Server Milestone 5.4.3 was released on October 9th, 2009 and was then followed up by 5.5.2 on Feb. 26, 2010. Initially there was a time of trying out and fine tuning the model before it found its current form around 5.5 GA. The stabilized model has been followed consistently throughout the 5.6 development cycle.   Our latest milestone release (5.7.1) was released this week and was first 5.7 milestone release. In total we have now done 11 milestones in 5.5, 5.6 and 5.7 together. In short we refer to these releases as DMR’s (Development Milestone Releases).

Why did we create the milestone release model? The milestone release model came into existence because our previous models failed badly. The problems were many: First, we were not able to ship on time (not even close), and in some cases not at all (maybe a blog for another day, “The MySQL releases that never shipped”). Second, quality was bad or unknown. Third, developers were frustrated. All these issues were fixed by the process change.

So what is the essence of the Milestone Release model? The heart of the milestone release model can be explained by three words: “mysql-trunk is sacred”. The key is to keep soundness and quality in our main development branch at all time. Two secrets here: One is to have massive regression testing on trunk at all times and to immediately fix anything that is broken. Second, to do massive QA on all new features before they are allowed into trunk. New features are developed in independent feature trees while frequently pulling in changes from trunk to keep it in sync. When feature development is done it goes to QA and will stay in QA until all bugs found by QA are fixed and a sign-off is given by QA. At this point, *after* QA is completed, the feature will be merged to trunk.

To summarize our experiences: With the Milestone Release model our historical problems are gone: A problematic feature will be stopped before it comes to trunk and will thus not pollute trunk. We are able to ship trunk at any time since quality is maintained on a daily basis. Developers are no longer blocked by a bad trunk, their feature goes in when it is ready.

This process is working out great internally. The process allows our enginners to focus on their work. However, the biggest benefit is to our customers and users as they are getting predictability, timeliness and quality.