MySQL Shell 8.0.25 for MySQL Server 8.0 and 5.7 has been released

Dear MySQL users,

MySQL Shell 8.0.25 is a maintenance release of MySQL Shell 8.0 Series (a
component of the MySQL Server). The MySQL Shell is provided under Oracle’s
dual-license.

MySQL Shell 8.0 is highly recommended for use with MySQL Server 8.0 and 5.7.
Please upgrade to MySQL Shell 8.0.25.

MySQL Shell is an interactive JavaScript, Python and SQL console interface,
supporting development and administration for the MySQL Server. It provides
APIs implemented in JavaScript and Python that enable you to work with MySQL
InnoDB Cluster and use MySQL as a document store.

The AdminAPI enables you to work with MySQL InnoDB Cluster and InnoDB
ReplicaSet, providing integrated solutions for high availability and
scalability using InnoDB based MySQL databases, without requiring advanced
MySQL expertise.  For more information about how to configure and work with
MySQL InnoDB Cluster and MySQL InnoDB ReplicaSet see

https://dev.mysql.com/doc/mysql-shell/8.0/en/admin-api-userguide.html

The X DevAPI enables you to create “schema-less” JSON document collections and
perform Create, Update, Read, Delete (CRUD) operations on those collections
from your favorite scripting language.  For more information about how to use
MySQL Shell and the MySQL Document Store support see

https://dev.mysql.com/doc/refman/en/document-store.html

For more information about the X DevAPI see

https://dev.mysql.com/doc/x-devapi-userguide/en/

If you want to write applications that use the the CRUD based X DevAPI you can
also use the latest MySQL Connectors for your language of choice. For more
information about Connectors see

https://dev.mysql.com/doc/index-connectors.html

For more information on the APIs provided with MySQL Shell see

https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/

and

https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/

Using MySQL Shell’s SQL mode you can communicate with servers using the legacy
MySQL protocol. Additionally, MySQL Shell provides partial compatibility with
the mysql client by supporting many of the same command line options.

For full documentation on MySQL Server, MySQL Shell and related topics, see

https://dev.mysql.com/doc/mysql-shell/8.0/en/

For more information about how to download MySQL Shell 8.0.25,
see the “General Availability (GA) Releases” tab at

http://dev.mysql.com/downloads/shell/

We welcome and appreciate your feedback and bug reports, see

http://bugs.mysql.com/

Enjoy!

Changes in MySQL Shell 8.0.25 (2021-05-11, General Availability)

Bugs Fixed


     * When the consistent option is set to true for MySQL
       Shell’s instance dump utility util.dumpInstance(), schema
       dump utility util.dumpSchemas(), and table dump utility
       util.dumpTables(), the utility locks the tables for
       backup during the dump. If the user ID used to run the
       utility has the RELOAD privilege, the utility uses a
       FLUSH TABLES WITH READ LOCK statement to set a global
       read lock. If the user ID does not have that privilege
       but does have the LOCK TABLES privilege, the utility
       issues a series of LOCK TABLES statements. Previously,
       the utility would issue these statements in the same
       session, causing previously locked tables to be unlocked.
       The utility now locks the system tables in the main
       thread’s session, and creates additional sessions to lock
       the other tables that are being dumped. (Bug #32788788)

     * MySQL Database Service validates that all user accounts
       have passwords, but MySQL Shell’s instance dump utility
       util.dumpInstance did not previously check this when the
       ocimds option was specified, resulting in errors when
       loading the dump files. The utility now includes this in
       the compatibility checks and returns an error if a user
       account does not have a password set, except where the
       user account is identified as a role. The
       skip_invalid_accounts modification, which can be applied
       using the compatibility option, now also removes user
       accounts that do not have passwords set. In the case of a
       user account that is identified as a role, the utility
       dumps the account using the CREATE ROLE statement. (Bug
       #32741098)

     * In some situations, MySQL Shell’s dump loading utility
       util.loadDump() did not delete the progress state file
       when the resetProgress option was specified. (Bug
       #32734880)

     * When MySQL Shell’s dump loading utility util.loadDump()
       was used to apply a dump that was still in the process of
       being created (with the waitDumpTimeout option), and
       pre-authenticated requests were used for the dump, an
       authorization error could be returned when fetching the
       .idx file for a table. The utility’s behavior when
       handling the manifest has now been refactored to create
       the correct handle to the .idx files and use the correct
       pre-authenticated request URL. (Bug #32734817)

Enjoy and thanks for the support!

On Behalf of the MySQL Engineering Team,
Balasubramanian Kandasamy

MySQL Shell 8.0.24 for MySQL Server 8.0 and 5.7 has been released

Dear MySQL users,

MySQL Shell 8.0.24 is a maintenance release of MySQL Shell 8.0 Series (a
component of the MySQL Server). The MySQL Shell is provided under Oracle’s
dual-license.

MySQL Shell 8.0 is highly recommended for use with MySQL Server 8.0 and 5.7.
Please upgrade to MySQL Shell 8.0.24.

MySQL Shell is an interactive JavaScript, Python and SQL console interface,
supporting development and administration for the MySQL Server. It provides
APIs implemented in JavaScript and Python that enable you to work with MySQL
InnoDB Cluster and use MySQL as a document store.

The AdminAPI enables you to work with MySQL InnoDB Cluster and InnoDB
ReplicaSet, providing integrated solutions for high availability and
scalability using InnoDB based MySQL databases, without requiring advanced
MySQL expertise.  For more information about how to configure and work with
MySQL InnoDB Cluster and MySQL InnoDB ReplicaSet see

https://dev.mysql.com/doc/mysql-shell/8.0/en/admin-api-userguide.html

The X DevAPI enables you to create “schema-less” JSON document collections and
perform Create, Update, Read, Delete (CRUD) operations on those collections
from your favorite scripting language.  For more information about how to use
MySQL Shell and the MySQL Document Store support see

https://dev.mysql.com/doc/refman/en/document-store.html

For more information about the X DevAPI see

https://dev.mysql.com/doc/x-devapi-userguide/en/

If you want to write applications that use the the CRUD based X DevAPI you can
also use the latest MySQL Connectors for your language of choice. For more
information about Connectors see

https://dev.mysql.com/doc/index-connectors.html

For more information on the APIs provided with MySQL Shell see

https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/

and

https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/

Using MySQL Shell’s SQL mode you can communicate with servers using the legacy
MySQL protocol. Additionally, MySQL Shell provides partial compatibility with
the mysql client by supporting many of the same command line options.

For full documentation on MySQL Server, MySQL Shell and related topics, see

https://dev.mysql.com/doc/mysql-shell/8.0/en/

For more information about how to download MySQL Shell 8.0.24, see the “General
Availability (GA) Releases” tab at

http://dev.mysql.com/downloads/shell/

We welcome and appreciate your feedback and bug reports, see

http://bugs.mysql.com/

Enjoy!

Changes in MySQL Shell 8.0.24 (2021-04-20, General Availability)


     * AdminAPI Added or Changed Functionality

     * AdminAPI Bugs Fixed

     * Functionality Added or Changed

     * Bugs Fixed

AdminAPI Added or Changed Functionality


     * The API command-line integration, used for scripting
       MySQL Shell, has been improved to use the function and
       option metadata to properly parse arguments and associate
       them to the corresponding parameter in the API call.
       Previously, the command-line integration was processing
       all the arguments as strings, causing errors using
       operations such as:

mysqlsh user@hostname:3306 — cluster setOption “autoRejoinTries”  5

       The command-line integration now interprets the data
       being passed in based on what the API functions expect to
       receive. This enhancement also introduces support for
       lists in command-line integration calls.
       Additionally, you can now access the MySQL Shell online
       help from the command-line integration. For example, to
       get help on the shell.options functions, issue:
$ mysqlsh — shell options –help

       For more information, see API Command Line Integration
(https://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-command-line-integration.html).
       In addition, when registering a new MySQL Shell extension
       function, the new boolean cli option is supported by the
       shell.addExtensionObjectMember() operation. When an
       operation is registered with the cli option set to true,
       the object and the functions are made available for the
       command-line integration. This enables you to extend the
       scripting possibilities of MySQL Shell. (Bug #31186637)

AdminAPI Bugs Fixed


     * Most of the AdminAPI operations contain metadata
       preconditions to determine if it is valid or not to
       execute them. When this check was being done, it was
       adding two entries to the log, one to indicate the check
       was about to be done, and another to indicate the
       metadata state. This meant that operations such as
       monitoring a cluster, which implies executing regular
       status requests, resulted in a large number of log
       entries. Now, these two log messages have been merged
       into a single entry that gets logged at the info level
       when the metadata state is not correct, and as debug info
       when the state is correct. In other words, if the
       metadata state is correct, the message is only logged
       when log_level=debug. (Bug #32582745)

     * The memberRole is now included in the default output of
       Cluster.status(). Previously, this information was only
       included when the extended option had a value of 1 or
       higher. This makes it easier to know what an instance’s
       role is in the cluster, regardless of whether its
       operating mode is R/W or R/O. (Bug #32381513)

     * dba.checkInstanceConfiguration() was performing an
       incorrect validation regarding the required privileges.
       This resulted in an endless loop where the operation
       detected missing privileges, you would give the grants as
       specified by the interactive help, but the operation
       would fail, indicating that the grants were missing. Now,
       the verification checks the correct list. Additionally,
       the internal list of required grants included SUPER,
       which is deprecated in 8.0. The fix replaces the SUPER
       grant with the fine-grained grants. (Bug #32287986)

     * The memberSslMode option did not support the VERIFY_CA
       and VERIFY_IDENTITY modes for the following operations:

          + dba.createCluster()

          + Cluster.addInstance()

          + Cluster.rejoinInstance()
       Now, the memberSslMode option supports these modes, and
       when they are used there is a validation to ensure that
       the CA certificates are supplied. If you choose to use
       the VERIFY_CA or VERIFY_IDENTITY mode, on each cluster
       instance you must manually supply the CA certificates
       using the ssl_ca and/or ssl_capath option. For more
       information, see Securing your Cluster
(https://dev.mysql.com/doc/mysql-shell/8.0/en/configuring-innodb-cluster.html#mysql-innodb-cluster-securing).
       Thanks to Daniël van Eeden for the contribution. (Bug
       #32247631, Bug #32241000)

     * In version 8.0.23, a check was added to verify if the
       server_id of all cluster instances is registered in the
       metadata as an instance attribute, and if not then the
       metadata is updated accordingly. This check is executed
       on add, rejoin and rescan operations. However, when
       upgrading a cluster from a version earlier than 8.0.23 to
       version 8.0.23 and higher, the server_id was not
       registered in the metadata unless you performed a manual
       rejoin. This was being silently ignored because it was
       not included in any diagnostic messages. Now, a new
       verification checks if the server_id of cluster instances
       is missing from the metadata and includes a note message
       in the instanceErrors attribute in the output of
       Cluster.status() indicating to use Cluster.rescan() to
       fix it. (Bug #32226871)

     * From MySQL Shell version 8.0.17, AdminAPI stores the
       replication or recovery accounts used for each added
       instance in the metadata schema, in the instances table.
       However, the specific transaction could fail or that
       entry might have been manually removed from the metadata
       schema, resulting in failures when trying to add other
       instances to the cluster, and there was no way to resolve
       this using AdminAPI. In such a situation, if you tried to
       add an instance the operation failed with an error. Now,
       AdminAPI tries to detect problems in the metadata related
       to these accounts for an InnoDB Cluster or InnoDB
       ReplicaSet. The status() operation prints a message if
       the required account is missing in the metadata schema,
       and also if it is not the one actually used. In such
       situations, the MySQL Shell help instructs you to either
       re-add the instance or run rescan() based on the detected
       problem. The addInstance() operation also prints a hint
       to call rescan() if any missing recovery users are found
       in the metadata. (Bug #32157182)

     * Cluster.addInstance() was permitting usage of the
       expelTimeout and consistency options when it should not.
       These options are cluster level settings that can only be
       set using dba.createCluster() and Cluster.setOption().
       (Bug #29779995)

     * The dba.checkInstanceConfiguration() operation detects if
       the instance has any tables that do not have a primary
       key. Group Replication requires every table that is to be
       replicated by the group to have a defined primary key.
       However, this does not mean that having a table without a
       primary key causes Group Replication to block or fail.
       Rather, the outcome is that changes to that table are not
       replicated but the group continues operating. Previously,
       if the dba.checkInstanceConfiguration() operation
       detected a table without a primary key, the operation
       returned with a status of ok and only mentioned tables
       with a missing primary key and unsupported engines. Now,
       if the operation detects such a table, it returns with a
       status of error. As part of this work, the
       dba.createCluster() operation has been changed to fail if
       it finds such tables. (Bug #29771457)

     * As part of the fix for Bug#28701263, AdminAPI started
       setting and persisting a default value of READ_ONLY for
       the group_replication_exit_state_action system variable.
       The default used by Group Replication was ABORT_SERVER.
       However, in MySQL Server 8.0.16 the default value of
       group_replication_exit_state_action became READ_ONLY so
       AdminAPI should not change and persist it. Now, on
       instances running 8.0.16 and later, the value of
       group_replication_exit_state_action is not modified. (Bug
       #29037274)

     * The exception information listed in the online help had
       become outdated and unwieldy for the interactive MySQL
       Shell, so it has been removed. (Bug #28542904)
       References: See also: Bug #29853828, Bug #32426083, Bug
       #32157120, Bug #28825389.

     * Using the allowRootFrom option with the
       dba.deploySandboxInstance() operation was creating a
       different remote root account depending on whether MySQL
       Shell was running in interactive mode or not. Now, the
       default value of allowRootFrom is consistent between both
       modes, and the account is created as root@% in both
       interactive and non-interactive mode. (Bug #27369121)

     * When you issue dba.createCluster() and
       dba.createReplica(), tables are created to store the
       metadata. If the default storage engine was not InnoDB,
       these operations could fail. Now, metadata creation
       operations always use the InnoDB storage engine. (Bug
       #101446, Bug #32110085)

Functionality Added or Changed


     * From MySQL 8.0.24, SQL statements that you issue in MySQL
       Shell’s SQL mode can be sent to the operating system’s
       system logging facility (syslog on Unix, or the Windows
       Event Log). You can select this option by specifying the
       –syslog command-line option when starting MySQL Shell,
       or by setting the history.sql.syslog MySQL Shell
       configuration option. SQL statements that would be
       excluded from the MySQL Shell code history are also
       excluded from the system logging facility. (Bug
       #31995742, Bug #31514599)

     * MySQL Shell’s instance dump utility util.dumpInstance(),
       schema dump utility util.dumpSchemas(), and table dump
       utility util.dumpTables() can now check for tables that
       do not contain primary keys. The check is carried out
       when the ocimds option is enabled for checking
       compatibility with MySQL Database Service, and an error
       is reported for every table included in the dump that
       does not have a primary key. The compatibility option,
       which implements appropriate measures for compatibility,
       has two new modification choices to notify MySQL Shell’s
       dump loading utility to create primary keys in invisible
       columns for tables that do not have them, or to ignore
       the missing primary keys. Primary keys are required for
       MySQL Database Service High Availability, which uses
       Group Replication.

Bugs Fixed


     * Previously, MySQL Shell retried requests to Oracle Cloud
       Infrastructure Object Storage a maximum of 5 times, with
       a 30 second wait in between retries, and a maximum
       overall wait of 5 minutes. The retry strategy has now
       been changed to increase the wait window and reduce the
       possibility of a dump or load operation failing. MySQL
       Shell now retries a maximum of 10 times, with a 1 minute
       wait in between retries, and a maximum overall wait of 10
       minutes. (Bug #32592962)

     * MySQL Shell’s instance dump utility util.dumpInstance()
       stopped with an error if the last schema to be dumped was
       a schema that contained no tables. The issue has now been
       fixed. (Bug #32540460)

     * MySQL Shell’s instance dump utility util.dumpInstance()
       has been optimized so that it can still be used
       successfully if there are limitations on the server’s
       resources such as disk space or the thread stack. To
       handle such situations, the queries from the utility can
       be repeated to retrieve smaller chunks of data if
       required, and file sorting is avoided. (Bug #32528186)

     * MySQL Shell’s instance dump utility util.dumpInstance()
       incorrectly removed grants of all privileges to users.
       The utility now expands GRANT ALL statements in the dump
       to list all privileges granted on all schemas and tables
       (*.*), and to list allowed privileges for system schemas.
       The dump loading utility util.loadDump() now extracts the
       lists of allowed and revoked global privileges during
       loading, and strips these from GRANT statements relating
       to system schemas and to all schemas and tables. (Bug
       #32526567)

     * MySQL Shell’s dump loading utility util.loadDump() now
       grants privileges after all the data is loaded.
       Previously, an error could occur if the utility tried to
       grant a privilege on a routine that did not yet exist.
       (Bug #32526496)

     * MySQL Shell’s instance dump utility util.dumpInstance(),
       schema dump utility util.dumpSchemas(), and table dump
       utility util.dumpTables() could not complete the dump if
       the gtid_executed system variable or the Information
       Schema’s COLUMN_STATISTICS table was unavailable. The
       utilities now display a warning message and log a
       detailed error message in this situation. These items are
       not required for a successful dump. (Bug #32515696)

     * MySQL Shell’s handling and formatting has been improved
       for the help text that you provide for dictionary
       parameters and their options when you register a Python
       plugin. (Bug #32509309)

     * MySQL Shell’s instance dump utility util.dumpInstance(),
       schema dump utility util.dumpSchemas(), and table dump
       utility util.dumpTables() no longer write a FLUSH TABLES
       statement to the binary log, as this can interfere with
       replication. (Bug #32490714)

     * From MySQL 8.0.23, MySQL Server supports replication from
       a source server that does not have GTIDs enabled and does
       not use GTID-based replication, to a replica that has
       GTIDs enabled, using the
       ASSIGN_GTIDS_TO_ANONYMOUS_TRANSACTIONS option of the
       CHANGE REPLICATION SOURCE TO statement. MySQL Shell’s
       instance dump utility util.dumpInstance(), schema dump
       utility util.dumpSchemas(), and table dump utility
       util.dumpTables() now support this functionality by
       storing the binary log file name and position in the dump
       metadata, in addition to the gtid_executed GTID set. The
       additional privilege REPLICATION CLIENT is required in
       order for the utilities to be able to collect this
       information, although if the user ID does not have that
       privilege, the dump continues but without the binary log
       information.
       The binary log information can be used after loading the
       dumped data into the replica server to set up replication
       with a non-GTID source server. MySQL Shell’s dump loading
       utility util.loadDump() prints the binary log and GTID
       set information from the dump metadata (in YAML format)
       when you specify the new option showMetadata: true. (Bug
       #32430402)

     * MySQL Shell did not correctly handle an empty array that
       was added to a collection. The result set normally
       returned from the server is now skipped in this
       situation. (Bug #32377134)

     * MySQL Shell’s instance dump utility util.dumpInstance(),
       schema dump utility util.dumpSchemas(), and table dump
       utility util.dumpTables() were unable to chunk table data
       from a MySQL 5.6 server instance, due to differences in
       the output of the EXPLAIN SELECT statement for that
       version. The utilities now account for the difference,
       and also cache the server version information for faster
       access. (Bug #32376447)

     * MySQL Shell’s parallel table import utility
       util.importTable() set a zero exit code if a non-critical
       error occurred that did not interrupt the import, such as
       a directory or file not being found. The utility now sets
       a non-zero error code instead when the first non-critical
       error is observed. (Bug #32286186)

     * MySQL Shell’s upgrade checker utility
       util.checkForServerUpgrade() now checks for spatial data
       columns that were originally created in MySQL 5.6. The
       underlying data type for such columns in MySQL 5.6 does
       not match their underlying data type in MySQL 8.0, so
       upgrade of the table is prohibited, and it must be
       recreated. (Bug #32257211, Bug #101944)

     * When MySQL Shell casts a string to a boolean value, the
       operation is now case insensitive. Previously, the
       results could differ between platforms. (Bug #32217910)

     * When MySQL Shell’s \warnings command was used to show
       warnings after each statement, warnings were not
       displayed for a classic MySQL protocol connection. (Bug
       #32151137)

     * MySQL Shell’s parallel table import utility
       util.importTable() now checks whether an uploaded object
       is a directory, and excludes these from wildcard matching
       that was specified for files. (Bug #31991122)

     * MySQL Shell’s dump loading utility util.loadDump() can
       split oversized chunks of data into smaller chunks for
       upload. Previously, if loading was stopped then resumed
       partway through this stage, the rows in the smaller
       chunks that were already loaded were not taken into
       account and skipped, which could lead to deadlocks. The
       utility’s progress file now records the smaller chunks
       individually so that they can be skipped if the load is
       stopped and resumed. (Bug #31961688)

     * An event that contained a sequence of two semi-colons
       caused MySQL Shell’s instance dump utility
       util.dumpInstance(), schema dump utility
       util.dumpSchemas(), and table dump utility
       util.dumpTables() to enter an infinite loop looking for
       delimiters. (Bug #31820571)

     * The decodeColumns option for MySQL Shell’s parallel table
       import utility util.importTable() could be specified
       without an accompanying columns option, resulting in the
       import stopping with an error. (Bug #31407058)

     * If a script that was run interactively in MySQL Shell’s
       Python mode did not have a newline character at the end,
       and the script ended with a multiline command, MySQL
       Shell waited for input instead of processing the command.
       The user had to press Enter to finish running the script,
       and the last line of the script was incorrectly saved in
       MySQL Shell’s code history. MySQL Shell now adds an empty
       line after processing a script input stream, to ensure
       that this situation does not occur. (Bug #30765725)

     * MySQL Shell used a different character set for collations
       depending on whether X Protocol or classic MySQL protocol
       was used to connect to the MySQL server instance, leading
       to inconsistency and in some situations, errors. For
       MySQL 5.7 instances, MySQL Shell now uses a SET NAMES
       statement to set all the relevant session system
       variables to the utf8mb4 character set. For MySQL 8.0
       instances, MySQL Shell now sets the collation_connection
       system variable to the utf8mb4_0900_ai_ci character set.
       (Bug #30516645)

Enjoy and thanks for the support!

On Behalf of the MySQL Engineering Team,
Nawaz Nazeer Ahamed

MySQL Shell 8.0.23 for MySQL Server 8.0 and 5.7 has been released

Dear MySQL users,

MySQL Shell 8.0.23 is a maintenance release of MySQL Shell 8.0 Series (a
component of the MySQL Server). The MySQL Shell is provided under
Oracle’s dual-license.

MySQL Shell 8.0 is highly recommended for use with MySQL Server 8.0 and
5.7. Please upgrade to MySQL Shell 8.0.23.

MySQL Shell is an interactive JavaScript, Python and SQL console
interface, supporting development and administration for the MySQL
Server. It provides APIs implemented in JavaScript and Python that
enable you to work with MySQL InnoDB cluster and use MySQL as a document
store.

The AdminAPI enables you to work with MySQL InnoDB cluster and InnoDB
ReplicaSet, providing integrated solutions for high availability and scalability
using InnoDB based MySQL databases, without requiring advanced MySQL
expertise.  For more information about how to configure and work with
MySQL InnoDB cluster and MySQL InnoDB ReplicaSet see

https://dev.mysql.com/doc/refman/en/mysql-innodb-cluster-userguide.html

The X DevAPI enables you to create “schema-less” JSON document
collections and perform Create, Update, Read, Delete (CRUD) operations
on those collections from your favorite scripting language.  For more
information about how to use MySQL Shell and the MySQL Document Store
support see

https://dev.mysql.com/doc/refman/en/document-store.html

For more information about the X DevAPI see

https://dev.mysql.com/doc/x-devapi-userguide/en/

If you want to write applications that use the the CRUD based X DevAPI
you can also use the latest MySQL Connectors for your language of
choice. For more information about Connectors see

https://dev.mysql.com/doc/index-connectors.html

For more information on the APIs provided with MySQL Shell see

https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/

and

https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/

Using MySQL Shell’s SQL mode you can communicate with servers using the
legacy MySQL protocol. Additionally, MySQL Shell provides partial
compatibility with the mysql client by supporting many of the same
command line options.

For full documentation on MySQL Server, MySQL Shell and related topics,
see

https://dev.mysql.com/doc/mysql-shell/8.0/en/

For more information about how to download MySQL Shell 8.0.23, see the
“General Availability (GA) Releases” tab at

http://dev.mysql.com/downloads/shell/

We welcome and appreciate your feedback and bug reports, see

http://bugs.mysql.com/

Enjoy and thanks for the support!

Changes in MySQL Shell 8.0.23 (2021-01-18, General Availability)

     * AdminAPI Added or Changed Functionality

     * AdminAPI Bugs Fixed

     * Functionality Added or Changed

     * Bugs Fixed

AdminAPI Added or Changed Functionality


     * The output of the status() operation has been extended to
       provide more information relevant to diagnosing errors.
       The following information is available for InnoDB
       Clusters and InnoDB ReplicaSets:

          + the memberState field shows the actual status of the
            instance as queried locally, which can be one of
            offline, error, recovering, or online.

          + a recovery.recoveryChannel field shows instances
            performing incremental recovery or in which the
            recovery channel status is not off

          + a new instanceErrors field exists for each instance,
            displaying any diagnostic information that can be
            detected for it

          + when the extended option is set to greater than 0,
            the output includes an applierChannel field, with
            replication information if the instance is either
            online and the applier channel status is not on, or
            the status is not recovering or online and the
            applier channel status is not off
       For more information, see Checking a cluster’s Status
       with Cluster.status()
       (https://dev.mysql.com/doc/mysql-shell/8.0/en/monitoring-
       innodb-cluster.html#check-innodb-cluster-status).

     * InnoDB Cluster and InnoDB ReplicaSet now support and
       enable parallel replication appliers, sometimes referred
       to as a multi-threaded replica. With the advances in
       MySQL such as binary log transaction dependency tracking
       and XXHASH64 based GTID set extraction, using multiple
       replica applier threads improves the throughput of both
       the replication applier and incremental recovery.
       This has resulted in the following changes:

          + the requirements for instances running 8.0.23 and
            later now also include:
               o binlog_transaction_dependency_tracking=WRITESET
               o slave_preserve_commit_order=ON
               o slave_parallel_type=LOGICAL_CLOCK
               o transaction_write_set_extraction=XXHASH64
            this means that new instances running 8.0.23 have
            these options configured by dba.configureInstance()
            and dba.configureReplicaSetInstance(). Attempting to
            add an instance running version 8.0.23 or later
            which does not have these variables configured
            results in an error. When you upgrade a cluster or
            replica set that has been running a version of MySQL
            server and MySQL Shell earlier than 8.0.23, the
            parallel replication applier is not enabled on the
            instances. This means you are not taking advantage
            of this feature, and you should reconfigure your
            instances to use the parallel replication applier.
            For more information, see Configuring the Parallel
            Replication Applier
            (https://dev.mysql.com/doc/mysql-shell/8.0/en/config
            uring-innodb-cluster.html#configuring-parallel-appli
            er).

          + dba.checkInstanceConfiguration() validates if
            parallel replication appliers are enabled or not.

          + the new applierWorkerThreads option configures the
            number of replication applier threads the instance
            uses for replication, and defaults to 4 threads. Use
            this option with the dba.configureInstance() and
            dba.configureReplicaSetInstance(). You can change
            this option while the instance is online, but the
            change is only made after the instance is restarted.

          + the output of the .status(extended=1) and options()
            operations now includes information about the
            configuration of parallel appliers.

AdminAPI Bugs Fixed


     * The fix for Bug#29305551 extended the
       dba.checkInstanceConfiguration() operation to include a
       check to verify if asynchronous replication is configured
       and running on the target instance, and print a warning
       when that is the case. This check is also used by the
       Cluster.addInstance() and Cluster.rejoinInstance()
       operations to terminate them with an error when such a
       scenario is detected, and is also used by the
       dba.rebootClusterFromCompleteOutage() operation whenever
       there are instances to be rejoined to the cluster.
       However, the dba.createCluster() operation was
       erroneously skipping the check, and the
       dba.rebootClusterFromCompleteOutage() operation was
       skipping the check on the instance being used to
       bootstrap the cluster. The fix ensures that the check is
       also performed whenever creating or rebooting a cluster
       from complete outage. Additionally, it adds support to
       override the check for the dba.createCluster() operation
       by making use of the force option, and it improves the
       error messages. (Bug #32197222)

     * The fix for Bug#29305551 extended
       dba.checkInstanceConfiguration() to verify if
       asynchronous replication is configured and running on the
       target instance and print a warning if that was the case.
       However, the check missed verifying if the replication
       channel was configured but not running. This fix ensures
       the verification also considers replication channels
       which are configured but are not actively running.
       Additionally, an erroneous message which suggested the
       possibility of using STOP REPLICA to override this check
       has been removed and replaced with an informative message
       which explains that unmanaged replication channels are
       not supported and the possible dangers of their usage.
       (Bug #32197197)

     * Based on the terminology changes in WL#14189
       (https://dev.mysql.com/worklog/task/?id=14189), AdminAPI
       has been aligned with the new terms. Error and log
       messages now use the terms source (previously master) and
       replica (previously slave). (Bug #32152133)

     * During a Cluster.rebootClusterFromCompleteOutage()
       operation, the GTID superset is used to detect which
       instance should be used to reboot the cluster. If an
       instance had a diverging GTID set and you wanted to
       explicitly remove it from the cluster, the operation
       blocked because it could not determine which instance had
       the GTID superset. Previously, in such a situation there
       was no way to exclude the instance from the instances
       used to detect the GTID superset. Now, if you answer no
       during the interactive wizard, or configure the
       removeInstances option, the instance is not checked as
       part of finding the GTID superset. (Bug #32112864)

     * When an instance had left a ReplicaSet, and then its
       configuration was changed in a way that made it invalid
       for InnoDB ReplicaSet usage, the
       ReplicaSet.rejoinInstance() operation did not detect that
       the configuration was invalid. Now, instances are checked
       to ensure they are valid before rejoining them to a
       ReplicaSet. (Bug #31975416)

     * When upgrading the metadata using dba.upgradeMetadata(),
       if there are MySQL Router instances that need to be
       upgraded, the operation waits until all instances are
       upgraded before continuing. The operation offers you an
       option to re-check for outdated MySQL Router instances
       and continue with the metadata upgrade. A MySQL Router
       upgrade is only complete after a restart of the
       application, however the message printed did not mention
       that. This message now includes the information that
       MySQL Router instances must be restarted after the
       binaries are upgraded. (Bug #31882876)

     * When you were connected to a secondary instance,
       attempting to issue operations such as
       Cluster.rejoinInstance(), Cluster.addInstance(),
       Cluster.dissolve() and so on would fail. Now, AdminAPI
       always connects to the current primary.
       As part of this work the following changes were made:

          + Now, in the event that dba.createCluster() or
            Cluster.addInstance() fail with a Group Replication
            error, AdminAPI returns the
            performance_schema.error_log entries.

          + The Cluster.rejoinInstance() operation has been
            changed to succeed if the instance is already in the
            cluster, instead of throwing an exception.

          + The dba.rebootCluster() operation has been changed
            to not clear super_read_only on the instance.
       (Bug #31757737)

     * As part of the default settings for InnoDB Cluster, to
       ensure that instances automatically rejoin the cluster,
       the group_replication_start_on_boot option is
       automatically set to true. However, this meant that in
       environments with an external tool managing the cluster
       life cycle, for example an orchestrator such as
       Kubernetes, the automatic enabling of rejoin could cause
       conflicts with the tool. In addition, if the automatic
       rejoining of an instance was enabled at an unsuitable
       time (for example when rebooting, or while repairing a
       split-brain, and so on), a deadlock or long freezes could
       occur until a timeout happened. In some situations,
       instances could even potentially join the wrong cluster
       during a reconfiguration.
       To avoid such situations, the manualStartOnBoot boolean
       option has been added, which defaults to false. To
       disable the automatic rejoining of an instance, for
       example while repairing a split-brain, set the
       manualStartOnBoot option to true. This prevents the
       instance rejoining the cluster automatically while you
       make changes. You then need to rejoin the instance to the
       cluster manually, before setting the manualStartOnBoot
       option back to false to ensure instance it rejoins the
       cluster automatically again. Similarly, if you are using
       an external orchestrator to manage the life cycle of
       instances, set the manualStartOnBoot option to true
       across the whole cluster, to disable the automatic
       rejoining of instances to the cluster. Your orchestrator
       should then be configured to rejoin the instances
       manually. (Bug #31643595)

     * Calling dba.checkInstanceConfiguration() with verifyMyCnf
       set to a file which did not exist, the operation
       completed successfully saying the configuration file had
       been checked. The fix checks if the file specified by
       verifyMyCnf exists, prints an error if not, and ensures
       the console does not show unnecessary error messages.
       (Bug #31468546)

     * On an instance with the sql_mode variable set to
       ANSI_QUOTES, attempting to upgrade the metadata schema
       with dba.upgradeMetadata() failed with the error: Unknown
       column ‘MySQL Router’ in ‘field list’. This was related
       to a query which uses single quotes to quote strings. As
       part of this fix, the upgrade metadata operation now
       prepares the session to be used by AdminAPI, and amongst
       other sanity checks it ensures that the sql_mode for that
       session uses the default value to avoid incompatible user
       configured settings. Additionally, the same was done for
       the dba.getCluster() and dba.dropMetadataSchema()
       operations. (Bug #31428813)

     * If the MySQL Shell global session was connected to a
       sandbox instance, and that instance was stopped, MySQL
       Shell tried to incorrectly reconnect to the instance.
       Now, if the active session is connected to a sandbox
       instance which is being stopped, MySQL Shell closes the
       session. (Bug #31113914)

     * The output of Cluster.status() now includes additional
       information about instances that are registered in the
       metadata but not currently online. MySQL Shell now
       connects to offline instances found in the metadata and
       attempts to diagnose them, providing additional
       information such as their connectivity and status. (Bug
       #30501615)

     * Instances that are part of the underlying group but are
       not identified in the metadata, for example because they
       were configured manually and bypassing MySQL Shell, or
       because they were previously removed from the InnoDB
       Cluster but were not properly decommissioned, are now
       shown in the output of Cluster.status(), along with
       diagnostic warnings about the metadata discrepancy. This
       ensures you can detect situations where an instance is
       participating in the group but is not being managed by
       MySQL Shell. (Bug #27882663)

     * An instance that belongs to an InnoDB Cluster is
       identified by its server UUID. If the UUID changed after
       the instance had left the cluster, for example because
       you used MySQL Enterprise Backup to restore from a
       backup, then the instance could not be rejoined to the
       cluster. Now, if the cluster encounters this situation,
       it checks the metadata to see if the instance can be
       identified using its host and port. If found, the
       metadata is updated based on the options used for the
       rejoin operation. This check is executed during the
       Cluster.rejoinInstance() and Cluster.rescan() operations.
       Additionally, a check is executed to verify the serverId
       of all the instances is registered in the metadata as an
       instance attribute. If it is not, the metadata is updated
       accordingly. This check is executed on add, rejoin and
       rescan operations. (Bug #26649039)

Functionality Added or Changed


     * MySQL Shell’s parallel table import utility can now
       import a specified list of input data files, and it
       supports wildcard pattern matching to include all
       relevant files from a location. Multiple files uploaded
       by a single run of the utility are placed into a single
       relational table, so for example, data that has been
       exported from multiple hosts and stored in multiple files
       could be merged into a single table to be used for
       analytics. The files can be compressed in the gzip or
       zstd format, and in that case the utility reads them from
       storage in the compressed format, saving bandwidth for
       that part of the transfer. The utility then uses its
       parallel connections to decompress and upload several
       files simultaneously to the target server.

Bugs Fixed


     * When MySQL Shell’s instance dump utility
       util.dumpInstance() was run with the ocimds option set to
       true to check compatibility with MySQL Database Service,
       and the users option set to true to include users and
       their roles and grants in the dump, the utility reported
       some compatibility errors for privileges that actually
       were permitted. MySQL Shell’s allowed list of privileges
       for MySQL Database Service has now been updated. (Bug
       #32213605)

     * The behavior of MySQL Shell’s table dump utility
       util.dumpTables() and dump loading utility
       util.loadDump() regarding the schemas for single table
       dumps and loads has been changed. Previously, the dump
       files produced for a single table did not contain the SQL
       statements to recreate the schema, so the schema had to
       exist in the target MySQL instance before the dump
       loading utility could load the table. Now, the dumps
       produced by the table dump utility contain the schema
       statements, and when they are loaded with the dump
       loading utility, by default, the schema is created in the
       target MySQL instance if it does not already exist. The
       schema option can be used to load the table dump into
       another schema that exists in the target MySQL instance.
       Table dumps created using the earlier version of the
       utility still require the schema option and an existing
       schema. (Bug #32165101)

     * MySQL Shell’s table dump utility util.dumpTables() now
       supports the ocimds, compatibility, ociParManifest, and
       ociParExpireTime options, so you can check compatibility
       with MySQL Database Service, and generate
       pre-authenticated request URLs for the dump files. Also,
       the ignoreVersion option has been extended to allow the
       import of a dump that was created without the ocimds
       option into a MySQL DB System. (Bug #32140970)

     * If a dump included users that were created with external
       authentication plugins, MySQL Shell’s dump loading
       utility util.loadDump() was unable to load the dump if
       those plugins were not available on the target server
       instance. The ocimds option for MySQL Shell’s instance
       dump utility util.dumpInstance() and schema dump utility
       util.dumpSchemas which checks compatibility with MySQL
       Database Service, now checks for accounts using
       authentication plugins that are not supported in MySQL
       Database Service. The compatibility option has an
       additional modification option skip_invalid_accounts,
       which removes such user accounts. (Bug #32115948)

     * Previously, MySQL Shell’s dump loading utility
       util.loadDump() stopped with an error if the loadUsers
       option was set to true but the supplied dump files did
       not contain user accounts. The utility now displays a
       warning and continues in this situation. (Bug #32115861)

     * MySQL Shell’s instance dump utility util.dumpInstance(),
       schema dump utility util.dumpSchemas(), and table dump
       utility util.dumpTables() falls back to using the LOCK
       TABLES privilege to lock dumped tables if the consistent
       option is set to true, which is the default, and the
       RELOAD privilege is not available. However, the locking
       operation could cause an implicit commit on active
       transactions, meaning that the data was not dumped
       consistently. The locking has now been corrected to
       ensure consistency in this situation. (Bug #32107327, Bug
       #101410)

     * When MySQL Shell’s dump loading utility util.loadDump()
       used indexes to identify row boundaries, an error
       occurred if an index pointed beyond the data in the read
       buffer. The utility now checks for this situation and
       ignores the index if so. (Bug #32072961)

     * When MySQL Shell was attempting to reconnect to a server,
       Ctrl + C did not interrupt the operation. The interrupt
       now functions and sets the retry attempts counter to zero
       so that the sequence exits correctly. (Bug #32041342)

     * MySQL Shell can now be built using Python 3.9. (Bug
       #32020230)

     * The updateGtidSet option for MySQL Shell’s dump loading
       utility util.loadDump() could not be used with MySQL DB
       System due to a permissions restriction. The utility now
       uses a stored procedure that is permitted, so the option
       can be used. (Bug #32009225)

     * When MySQL Shell’s instance dump utility
       util.dumpInstance(), schema dump utility
       util.dumpSchemas(), or table dump utility
       util.dumpTables() was exporting to an Oracle Cloud
       Infrastructure Object Storage bucket, if there was a loss
       of connectivity or routing to the Object Storage server,
       MySQL Shell stopped unexpectedly. The error is now
       handled correctly. (Bug #32005418)

     * MySQL Shell’s dump loading utility util.loadDump()
       returned an exception if a header value in a response was
       empty. (Bug #31979374)

     * MySQL Shell did not initialize Python 3.8’s new
       cf_feature_version compiler flag field, which could cause
       an exception when format strings were used. (Bug
       #31926697)

     * Where MySQL Shell is using a system installation of
       Python rather than the bundled version, the minimum
       version that MySQL Shell supports is now Python 3.6.
       Python 3.4.3 was the previous minimum for a system
       installation. The bundled version is Python 3.7.7. (Bug
       #31900744)

     * MySQL Shell’s instance dump utility util.dumpInstance(),
       schema dump utility util.dumpSchemas(), and table dump
       utility util.dumpTables() use table statistics to
       identify a suitable default row size. If the statistics
       for a table are outdated or not present, this can cause
       issues for the chunking process. In this situation, MySQL
       Shell now issues a message to suggest using an ANALYZE
       TABLE statement to produce up to date statistics. (Bug
       #31766490)

     * The skipBinlog option for MySQL Shell’s dump loading
       utility util.loadDump() skips binary logging on the
       target MySQL instance for the import. The option is not
       suitable for MySQL DB System as the binary logging status
       cannot be changed, and the import now fails with an error
       message if the option is used in that situation. For
       other MySQL instances, the utility now checks whether the
       user has the required privileges to set the sql_log_bin
       system variable, and fails with an error message if they
       do not. (Bug #31748786)

     * MySQL Shell’s instance dump utility util.dumpInstance(),
       schema dump utility util.dumpSchemas(), and table dump
       utility util.dumpTables() ordered the data fetched for
       export using the first column of a unique index for the
       table. The same method was used to query data for
       chunking purposes. The utilities now use all columns of
       the unique index for ordering. In addition, performance
       is improved by the addition of a cache to store
       frequently-used instance metadata. The cache is populated
       for all the schema objects at once, rather than by
       individual queries as needed. (Bug #31706755)

     * MySQL Shell’s disconnect function was added to the shell
       global object. (Bug #31704380)

On Behalf of Oracle/MySQL Release Engineering
Prashant Tekriwal

MySQL Shell 8.0.22 for MySQL Server 8.0 and 5.7 has been released

Dear MySQL users,

MySQL Shell 8.0.22 is a maintenance release of MySQL Shell 8.0 Series (a component of the MySQL Server). The MySQL Shell is provided under Oracle’s dual-license.

MySQL Shell 8.0 is highly recommended for use with MySQL Server 8.0 and 5.7. Please upgrade to MySQL Shell 8.0.22.

MySQL Shell is an interactive JavaScript, Python and SQL console interface, supporting development and administration for the MySQL Server. It provides APIs implemented in JavaScript and Python that enable you to work with MySQL InnoDB cluster and use MySQL as a document store.

The AdminAPI enables you to work with MySQL InnoDB cluster and InnoDB ReplicaSet, providing integrated solutions for high availability and scalability using InnoDB based MySQL databases, without requiring advanced MySQL expertise. For more information about how to configure and work with MySQL InnoDB cluster and MySQL InnoDB ReplicaSet see

https://dev.mysql.com/doc/refman/en/mysql-innodb-cluster-userguide.html

The X DevAPI enables you to create “schema-less” JSON document collections and perform Create, Update, Read, Delete (CRUD) operations on those collections from your favorite scripting language. For more information about how to use MySQL Shell and the MySQL Document Store support see

https://dev.mysql.com/doc/refman/en/document-store.html

For more information about the X DevAPI see

https://dev.mysql.com/doc/x-devapi-userguide/en/

If you want to write applications that use the the CRUD based X DevAPI you can also use the latest MySQL Connectors for your language of choice. For more information about Connectors see

https://dev.mysql.com/doc/index-connectors.html

For more information on the APIs provided with MySQL Shell see

https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/

and

https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/

Using MySQL Shell’s SQL mode you can communicate with servers using the legacy MySQL protocol. Additionally, MySQL Shell provides partial compatibility with the mysql client by supporting many of the same command line options.

For full documentation on MySQL Server, MySQL Shell and related topics,
see

https://dev.mysql.com/doc/mysql-shell/8.0/en/

For more information about how to download MySQL Shell 8.0.22, see the
“General Availability (GA) Releases” tab at

http://dev.mysql.com/downloads/shell/

We welcome and appreciate your feedback and bug reports, see

http://bugs.mysql.com/

Enjoy and thanks for the support!

Changes in MySQL Shell 8.0.22 (2020-10-19, General Availability)

  • AdminAPI Bugs Fixed
  • Functionality Added or Changed
  • Bugs Fixed

AdminAPI Bugs Fixed

  • Due to the changes introduced in bug#31467823, AdminAPI user messages have been updated to use the same terminology. (Bug #31798724) References: See also: Bug #31462783, Bug #31467823.
  • In 8.0.22, Group Replication deprecates the group_replication_ip_whitelist system variable in favor of group_replication_ip_allowlist. Therefore AdminAPI now includes a new option named ipAllowlist and deprecates the ipWhitelist option used with dba.createCluster(), Cluster.addInstance() and Cluster.rejoinInstance(). Regardless of the target instance version, if ipWhitelist is used MySQL Shell prints a deprecation warning. When the target instance is 8.0.22 or later Group Replication
    reports a deprecation warning, but AdminAPI avoids this by ensuring the appropriate variable is set according to the version of MySQL running on the instance. If the target instance is 8.0.22 or later and the old ipWhitelist option is used, AdminAPI updates the new group_replication_ip_allowlist system variable. On instances running MySQL 8.0.21 or earlier, AdminAPI updates the older group_replication_ip_whitelist system variable.
    This work also fixes a bug in Cluster.options(), which shows the list of all Group Replication options that are configurable by MySQL Shell. When an option is not set or does not exist (for example in a certain version) it is displayed as having a null value. However, when you passed in the all option to Cluster.options(), such variables were being excluded from the list. (Bug#31798495)
  • The dba.configureLocalInstance() and dba.configureInstance() operations could not be used against instances that were part of unmanaged replication groups. This made it impossible to create a cluster administrator account, which is required when adopting a Group Replication group into an InnoDB Cluster. Creating cluster administrator accounts is very important for the remote management of InnoDB Clusters, and to avoid having to manually create the user and required privileges on
    each instance. The fix ensures that dba.configureLocalInstance() and dba.configureInstance() can be executed against instances belonging to unmanaged
    replication groups. Because such instances are ready for InnoDB Cluster usage, a message confirms this. (Bug#31691232)
  • The dba.rebootClusterFromCompleteOutage() operation was only checking GTID_EXECUTED when validating for the instance that has the most transactions. Transactions that were received (and certified) but not executed yet were not included in that check. Also, transactions received through replication channels other than the Group Replication applier were also not being considered in that check, because the instance was probably the primary while the cluster was running. Ignoring these transactions led to data loss. Now, any known and managed
    replication channels are considered as part of the check. (Bug #31673163)
  • The dba.removeInstance() operation failed if the instance being removed was not only unreachable but also unresolvable, which could be the case when the instance
    was running in a container that takes down its own DNS record when removed.
    Instances could not be removed even if force was enabled, because of a validation that ensured that the given instance is a valid address by checking if it is
    resolvable. If that validation failed, nothing else was attempted. The fix completely removes address resolution from the whole AdminAPI. That check was redundant,
    because invalid addresses would eventually lead to an error anyway when a connection was opened. IPv6 address syntax validations were left to remind you that ::1 should be specified as [::1]. (Bug #31632606)
  • Calling Cluster.rejoinInstance(“host:port”) with no user name specified caused AdminAPI to try and connect using the operating system user name instead of the credentials used to connect the cluster object. Now, if credentials are not provided, they are taken from the target server’s connection options. (Bug #31632554)
  • When adding an instance to a cluster using MySQL Clone and monitoring the transfer of data, MySQL Shell could stop unexpectedly. This was due to the assumption that the Performance Schema would provide information on all four stages of cloning, which might not exist if the data set was very small. The fix ensures that updates are performed according to the information which is available
    at each time. (Bug #31545728)
  • When an unreachable primary instance was forcibly removed from a cluster, the group_replication_group_seeds system variable was not updated because the group status was queried from the primary, which was missing. The cluster was left in an inconsistent state, and if any of the instances were restarted, they would not be able to automatically rejoin, because group_replication_group_seeds contained an invalid
    address, which caused Group Replication to abort trying to join the group.
    (Bug #31531704)
  • If the validate_password plugin was enabled, the setupAdminAccount() and setupRouterAccount() operations would fail with an error indicating the password did not meet the policy requirements. This happened regardless of whether the password met the policy requirements or not. (Bug #31491092)
  • In the event of dba.createCluster() failing, the metadata record was left behind and Group Replication was also started. The cluster was left in an inconsistent state
    that could not be recovered from when calling dba.createCluster() again. Now, metadata changes during a dba.createCluster() operation are enclosed in a
    transaction, so that both the cluster and instance records are only committed if the operation succeeds. Group Replication is also stopped if the create operation
    does not complete successfully. Similarly, Cluster.addInstance() has been changed to ensure that its metadata record is only inserted when everything else has completed successfully. On retry, it skips any steps that would cause conflicts because Group Replication is already running. This introduces a behavior change, where adding an instance that is part of the group but not in the metadata just adds it to the metadata, instead of aborting and requiring you to execute Cluster.rescan(). (Bug #31455419)
  • If a connection timed out during a Cluster.status() operation, MySQL Shell could appear to hang for a long time. To improve the responsiveness, the default timeout
    of AdminAPI operations has been reduced from 10 seconds to 2 seconds. This ensures operations like Cluster.status() do not appear to freeze for a long time
    when there are unreachable instances. (Bug #30884174)
  • Instances operating in an InnoDB Cluster or InnoDB ReplicaSet are all required to have the same password for the administrative account. In a situation where the
    password on an instance joining a InnoDB Cluster or InnoDB ReplicaSet did not match the other instances, the error message did not explain this and the instance
    failed to join. Now, in such a situation the error is detected and the resulting message mentions that the password is the cause of the failure. It is recommended
    that you set up administrator accounts using the setupAdminAccount() operation, see Creating User Accounts for Administration (https://dev.mysql.com/doc/refman/8.0/en/admin-api-overvi ew.html#creating-user-accounts-for-admin-api).
    (Bug#30728744)

Functionality Added or Changed

  • Two new utilities are available in MySQL Shell to export single tables from a MySQL server instance.
    • MySQL Shell’s new table dump utility util.dumpTables() supports the export of a selection of tables or views from a schema, from an on-premise
      MySQL instance into an Oracle Cloud Infrastructure Object Storage bucket or a set of local files. It works in the same way as the instance dump utility
      util.dumpInstance() and schema dump utility util.dumpSchemas() introduced in 8.0.21, but with a different selection of suitable options. The exported items can then be imported into a MySQL Database Service DB System (a MySQL DB System, for short) or a MySQL Server instance using MySQL Shell’s dump loading utility util.loadDump().
    • MySQL Shell’s new table export utility util.exportTable() exports a MySQL relational table into a data file in a variety of formats, either on the local server or in an Oracle Cloud Infrastructure Object Storage bucket. The data can
      then be uploaded into a table on a target MySQL server using MySQL Shell’s parallel table import utility util.importTable(), which uses parallel connections to provide rapid data import for large data files. The data file can also be used to import data to a different application, or as a lightweight logical backup for a single data table.
  • From MySQL Shell 8.0.22, when you export schemas to an Oracle Cloud Infrastructure Object Storage bucket using MySQL Shell’s instance dump utility and schema dump utility, util.dumpInstance() and util.dumpSchemas(), during the dump you can generate a pre-authenticated request URL for every item. The utilities do this by default when the ocimds option is set to true, and you can control the feature using the ociParManifest and ociParExpireTime options. The user account that runs MySQL Shell’s dump loading utility util.loadDump() then uses the pre-authenticated request URLs to load the dump files without additional access permissions.

Bugs Fixed

  • MySQL Shell’s dump loading utility util.loadDump() would stop with an error if a data file’s size was larger than an applicable server limit relating to the maximum
    transaction size, such as the max_binlog_cache_size limit. Now, the utility stops the data load in mid-file if the number of bytes uploaded is about to exceed 1.5 times the bytesPerChunk setting of the utility that created the data files, which is stored in the dump metadata. The data load is then restarted to upload the remainder of the file. Due to this new safeguard, the default bytesPerChunk setting of MySQL Shell’s instance dump utility util.dumpInstance(), schema dump utility util.dumpSchemas(), and table dump utility util.dumpTables() has been increased from 32 MB to 64 MB.
    (Bug #31945539)
  • MySQL Shell’s instance dump utility util.dumpInstance() and schema dump utility util.dumpSchemas() did not take the row size into account when deciding whether to chunk table data, only the number of rows. A table with a smaller number of rows containing large amounts of data might therefore bypass chunking even if the row size exceeded the specified chunk size for the dump. The utilities now carry out chunking regardless of the number of rows, unless the data is estimated to fit in a single chunk of the specified size. (Bug #31938831)
  • When MySQL Shell’s instance dump utility util.dumpInstance() or schema dump utility util.dumpSchemas() was splitting a table into chunks, an integer overflow, loop, and consequent out of memory error could occur if the maximum value in the index
    column was close to the maximum value of an integer. Extra checks have now been added to avoid this situation. (Bug #31896448)
  • Corrupted SQL files could occur in MySQL Shell’s dumps if the dumped string was exactly 2048 bytes, due to an internal module error which is now accounted for.
    (Bug#31843832)
  • The administrative user account on an Oracle Cloud Infrastructure Compute instance, which was used when importing data with MySQL Shell’s dump loading utility util.loadDump(), was unable to revoke privileges on MySQL system schemas (mysql and sys) that it did not have itself. Affected REVOKE statements are now stripped from an instance or schema dump created by MySQL Shell’s instance dump utility util.dumpInstance() and schema dump utility util.dumpSchemas() when the strip_restricted_grants compatibility option is used. (Bug #31842532)
  • MySQL Shell’s instance dump utility and schema dump utility, dumpInstance() and dumpSchemas(), previously fetched the value of the gtid_executed system variable
    before the read lock was established, which could potentially lead to an inconsistency with the dumped data. The GTID set is now retrieved while the read lock is active. (Bug #31706940)
  • From MySQL Shell 8.0.22, you can use the -pym command-line option to execute a specified Python module as a script in MySQL Shell’s Python mode. –pym works in
    the same way as Python’s -m command line option. (Bug#31694202)
  • In MySQL Shell in Python mode, functions registered in JavaScript could not be called from global extension objects if they had optional arguments that were not
    provided. (Bug #31693096)
  • MySQL Shell treated the character sequence */ as the end of a comment even when it was part of a quoted string. (Bug #31689135)
  • MySQL Shell’s instance dump utility and schema dump utility, dumpInstance() and dumpSchemas(), automatically select an index column to order and chunk the data. When an index with a functional key part was present for a table, the query for index columns returned NULL for the column name, causing an exception in the utility. These column names are now filtered out in the query. (Bug#31687059)
  • MySQL Shell’s parallel table import utility util.importTable() has a new option decodeColumns, and an enhancement to the columns option, to enable you to
    capture columns from the import file for input preprocessing (or to discard them) in the same way as with a LOAD DATA statement. The decodeColumns option
    specifies preprocessing transformations for the captured data in the same way as the SET clause of a LOAD DATA statement, and assigns them to columns in the target table. (Bug #31683641)
  • MySQL Shell’s dump loading utility loadDump() now verifies that it can open and write to the progress state file before it starts to retrieve the dump files, so that
    the utility does not spend time fetching the dump files if the import is subsequently going to fail for that reason. (Bug #31667539)
  • Before MySQL 8.0, user accounts that have all privileges used the statement GRANT ALL PRIVILEGES, rather than the full list of privileges. When MySQL Shell’s instance dump utility dumpInstance() was used to dump an instance, ALL PRIVILEGES was stripped from the statement, leaving the accounts with no privileges. The utility now replaces this grant with the administrator role.
    (Bug #31661180)
  • MySQL Shell’s instance dump utility and schema dump utility, dumpInstance() and dumpSchemas(), and dump loading utility loadDump(), previously timed out after
    the main thread had been idle for 8 hours. The timeout has now been extended indefinitely (to 1 year) so that long data load times do not cause the dump or import to fail. (Bug #31652265)
  • MySQL Shell’s dump loading utility loadDump() now includes a comment in its LOAD DATA statements to identify the data chunk that is currently being loaded.
    If you need to cancel the import, you can use this information to decide whether to let the import of this chunk complete or stop it immediately. (Bug #31646650)
  • Previously, MySQL Shell’s dump loading utility loadDump() closed DDL files after all tables from the schema had finished loading. For a dump with a very large number of DDL files, this could lead to unexplained runtime errors being returned. The utility now closes DDL files immediately after reading them. (Bug #31645896)
  • MySQL Shell’s dump loading utility loadDump() previously used only its main thread to execute the DDL scripts for tables. For a dump containing a large number of tables, fetching the DDL scripts could have a significant impact on the time taken. The utility now fetches and executes DDL scripts for tables using all its threads, with the exception of DDL scripts for views, which are fetched in parallel but executed only in the main thread to avoid race conditions. (Bug #31645806)
  • MySQL Shell’s dump loading utility loadDump() now loads views in sequence and only after all tables and placeholders from all schemas have been loaded, ensuring
    that views do not reference items that do not yet exist. (Bug #31645792)
  • MySQL Shell now ignores any other available Python installations when a bundled compatible version of the Python interpreter has been installed. (Bug #31642521)
  • When MySQL Shell’s dump loading utility loadDump() is importing users and their roles and grants, an error is now returned if the user already exists in the target
    instance, and the user’s grants from the dump files are not applied. Previously, the grants were applied to the existing user. (Bug #31627432)
  • MySQL Shell’s dump loading utility loadDump() now has an option updateGtidSet to apply the gtid_executed GTID set from the source MySQL instance to the gtid_purged GTID set on the target MySQL instance. You can append or replace the GTID set depending on the release of the target MySQL instance. (Bug #31627419)
  • MySQL Shell’s instance dump utility dumpInstance() and dump loading utility loadDump() now have options to include (includeUsers) or exclude (excludeUsers ) named user accounts from the dump files or from the import. You can use these options to exclude user accounts that are not accepted for import to a MySQL DB System, or that already exist or are not wanted on the target MySQL instance.
    (Bug #31627292)
  • With the deferTableIndexes option set to all, MySQL Shell’s dump loading utility loadDump() defers creation of all secondary indexes until after the table is loaded.
    Previously, a table with a unique key column containing an auto-increment value failed to load in this situation. The utility now also creates indexes defined on columns with auto-increment values when the deferTableIndexes option is set to all. (Bug #31602690)
  • The upload method used by MySQL Shell’s instance dump utility util.dumpInstance() and schema dump utility util.dumpSchemas() to transfer files to an Oracle Cloud
    Infrastructure Object Storage bucket has a file size limit of 1.2 TiB. In MySQL Shell 8.0.21, the multipart size setting means that the numeric limit on multiple file parts applies first, creating a limit of approximately 640 GB. From MySQL Shell 8.0.22, the
    multipart size setting has been changed to allow the full file size limit.
    (Bug #31589858)
  • MySQL Shell’s upgrade checker utility checkForServerUpgrade() now checks for the obsolete NO_AUTO_CREATE_USER SQL mode. (Bug #31501981, Bug #99903)
  • The parallelization of table loading by MySQL Shell’s dump loading utility loadDump() has been improved. (Bug#31441903)
  • If the settings for the server’s global character set variables differed from the settings for the current session, MySQL Shell’s parallel table import utility importTable() could not import data into a table created in the session whose name contained non-ASCII characters. Now, when the utility’s characterSet option is specified, MySQL Shell executes a SET NAMES statement with the given value. (Bug #31412330)
  • MySQL Shell’s parallel table import utility importTable() could not import data if the global SQL mode NO_BACKSLASH_ESCAPES was set. The utility now clears the
    global SQL mode in sessions created to run the import. (Bug #31407133)
  • When a function that was defined as a member of a MySQL Shell extension object had a number of optional parameters but no required parameters, in some situations
    calling the function with zero parameters or one parameter returned an error.
    (Bug #30744994)
  • MySQL Shell’s db global object did not return the current schema when its properties were queried. (Bug #30296825, Bug #96839)

On Behalf of Oracle/MySQL Release Engineering Team,
Prashant Tekriwal


MySQL Shell 8.0.21 for MySQL Server 8.0 and 5.7 has been released

Dear MySQL users,

MySQL Shell 8.0.21 is a maintenance release of MySQL Shell 8.0 Series (a
component of the MySQL Server). The MySQL Shell is provided under
Oracle’s dual-license.

MySQL Shell 8.0 is highly recommended for use with MySQL Server 8.0 and
5.7. Please upgrade to MySQL Shell 8.0.21.

MySQL Shell is an interactive JavaScript, Python and SQL console
interface, supporting development and administration for the MySQL
Server. It provides APIs implemented in JavaScript and Python that
enable you to work with MySQL InnoDB cluster and use MySQL as a document
store.

The AdminAPI enables you to work with MySQL InnoDB cluster and InnoDB
ReplicaSet, providing integrated solutions for high availability and scalability
using InnoDB based MySQL databases, without requiring advanced MySQL
expertise.  For more information about how to configure and work with
MySQL InnoDB cluster and MySQL InnoDB ReplicaSet see

https://dev.mysql.com/doc/refman/en/mysql-innodb-cluster-userguide.html

The X DevAPI enables you to create “schema-less” JSON document
collections and perform Create, Update, Read, Delete (CRUD) operations
on those collections from your favorite scripting language.  For more
information about how to use MySQL Shell and the MySQL Document Store
support see

https://dev.mysql.com/doc/refman/en/document-store.html

For more information about the X DevAPI see

https://dev.mysql.com/doc/x-devapi-userguide/en/

If you want to write applications that use the the CRUD based X DevAPI
you can also use the latest MySQL Connectors for your language of
choice. For more information about Connectors see

https://dev.mysql.com/doc/index-connectors.html

For more information on the APIs provided with MySQL Shell see

https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/

and

https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/

Using MySQL Shell’s SQL mode you can communicate with servers using the
legacy MySQL protocol. Additionally, MySQL Shell provides partial
compatibility with the mysql client by supporting many of the same
command line options.

For full documentation on MySQL Server, MySQL Shell and related topics,
see

https://dev.mysql.com/doc/mysql-shell/8.0/en/

For more information about how to download MySQL Shell 8.0.21, see the
“General Availability (GA) Releases” tab at

http://dev.mysql.com/downloads/shell/

We welcome and appreciate your feedback and bug reports, see

http://bugs.mysql.com/

Enjoy and thanks for the support!

Changes in MySQL Shell 8.0.21 (2020-07-13, General Availability)

     * AdminAPI Added or Changed Functionality
     * AdminAPI Bugs Fixed
     * Functionality Added or Changed
     * Bugs Fixed

AdminAPI Added or Changed Functionality

     * A new user configurable tag framework has been added to
       the metadata, to allow specific instances of a cluster or
       ReplicaSet to be marked with additional information. Tags
       can be any ASCII character and provide a namespace. You
       set tags for an instance using the setInstanceOption()
       operation. In addition, AdminAPI and MySQL Router 8.0.21
       support specific tags, which enable you to mark instances
       as hidden and remove them from routing. MySQL Router then
       excludes such tagged instances from the routing
       destination candidates list. This enables you to safely
       take a server instance offline, so that applications and
       MySQL Router ignore it, for example while you perform
       maintenance tasks, such as server upgrade or
       configuration changes. To bring the instance back online,
       use the setInstanceOption() operation to remove the tags
       and MySQL Router adds the instance back to the routing
       destination candidates list, and it becomes online for
       applications. For more information, see Tagging the
       Metadata
(https://dev.mysql.com/doc/refman/8.0/en/admin-api-tagging.html).

AdminAPI Bugs Fixed

     * Important Change: Previously, Group Replication did not
       support binary log checksums, and therefore one of the
       requirements for instances in InnoDB cluster was that
       binary log checksums were disabled by having the
       binlog_checksum system variable set to NONE. AdminAPI
       verified the value of binlog_checksum during the
       dba.checkInstanceConfiguration() operation and disallowed
       creating a cluster or adding an instance to a cluster
       that did not have binary log checksums disabled. In
       version 8.0.21, Group Replication has lifted this
       restriction, therefore InnoDB cluster now permits
       instances to use binary log checksums, with
       binlog_checksum set to CRC32. The setting for
       binlog_checksum does not have to be the same for all
       instances. In addition, sandboxes deployed with version
       8.0.21 and later do not set the binlog_checksum variable,
       which defaults to CRC32. (Bug #31329024)

     * Adopting a Group Replication setup as a cluster can be
       performed when connected to any member of the group,
       regardless of whether it is a primary or a secondary.
       However, when a secondary member was used,
       super_read_only was being incorrectly disabled on that
       instance. Now, all operations performed during an
       adoption are done using the primary member of the group.
       This ensures that no GTID inconsistencies occur and that
       super_read_only is not incorrectly disabled on secondary
       members. (Bug #31238233)

     * Using the clusterAdmin option to create a user which had
       a netmask as part of the host resulted in an error when
       this user was passed to the dba.createCluster()
       operation. Now, accounts that specify a netmask are
       treated as accounts with wildcards, meaning that further
       checks to verify if the account accepts remote
       connections from all instances are skipped. (Bug
       #31018091)

     * The check for instance read-only compatibility was using
       a wrong MySQL version as the base version. The
       cross-version policies were added to Group Replication in
       version 8.0.17, but the check was considering instances
       running 8.0.16. This resulted in a misleading warning
       message indicating that the added instance was read-only
       compatible with the cluster, when this was not true (only
       for instances 8.0.16). The fix ensures that the check to
       verify if an instance is read-compatible or not with a
       cluster is only performed if the target instance is
       running version 8.0.17 or later. (Bug #30896344)

     * The maximum number of instances in an InnoDB cluster is
       9, but AdminAPI was not preventing you from trying to add
       more instances to a cluster and the resulting error
       message was not clear. Now, if a cluster has 9 instances,
       Cluster.addInstance prevents you adding more instances.
       (Bug #30885157)

     * Adding an instance with a compatible GTID set to a InnoDB
       cluster or InnoDB ReplicaSet on which provisioning is
       required should not require any interaction, because this
       is considered a safe operation. Previously, in such a
       scenario, when MySQL Clone was supported MySQL Shell
       still prompted to choose between cloning or aborting the
       operation. Now, the operation proceeds with cloning,
       because this is the only way to provision the instance.
       Note
       instances with an empty GTID set are not considered to
       have a compatible GTID set when compared with the InnoDB
       cluster or InnoDB ReplicaSet. Such scenarios are
       considered to be unknown, therefore MySQL Shell prompts
       to confirm which action should be taken.
       (Bug #30884590)

     * The Group Replication system variables (prefixed with
       group_replication) do not exist if the plugin has not
       been loaded. Even if the system variables are persisted
       to the instance’s option file, they are not loaded unless
       the Group Replication plugin is also loaded when the
       server starts. If the Group Replication plugin is
       installed after the server starts, the option file is not
       reloaded, so all system variables have default values.
       Instances running MySQL 8.0 do not have a problem because
       SET PERSIST is used. However, on instances running
       version MySQL 5.7, the dba.rebootCluster() operation
       could not restore some system variables if the Group
       Replication plugin was uninstalled. Now, the
       dba.configureInstance() operation persists the Group
       Replication system variables to configuration files with
       the loose_ prefix. As a result, once the Group
       Replication plugin is installed, on instances running 5.7
       the persisted values are used instead of the default
       values. (Bug #30768504)

     * The updateTopologyMode option has been deprecated and the
       behavior of Cluster.rescan() has been changed to always
       update the topology mode in the Metadata when a change is
       detected. MySQL Shell now displays a message whenever
       such a change is detected. (Bug #29330769)

     * The cluster.addInstance() and cluster.rejoinInstance()
       operations were not checking for the full range of
       settings which are required for an instance to be valid
       for adding to the cluster. This resulted in attempts to
       use instances which run on different operating systems to
       fail. For example, a cluster running on two instances
       that were hosted on a Linux based operating system would
       block the addition of an instance running Microsoft
       Windows. Now, cluster.addInstance() and
       cluster.rejoinInstance() operations validate the instance
       and prevent adding or rejoining an instance to the
       cluster if the value of the lower_case_table_names,
       group_replication_gtid_assignment_block_size or
       default_table_encryption of the instance are different
       from the ones on the cluster. (Bug #29255212)

Functionality Added or Changed

     * MySQL Shell now has an instance dump utility,
       dumpInstance(), and schema dump utility, dumpSchemas().
       The new utilities support the export of all schemas or a
       selected schema from an on-premise MySQL server instance
       into an Oracle Cloud Infrastructure Object Storage bucket
       or a set of local files. The schemas can then be imported
       into a MySQL Database Service DB System using MySQL
       Shell’s new dump loading utility. The new utilities
       provide Oracle Cloud Infrastructure Object Storage
       streaming, MySQL Database Service compatibility checks
       and modifications, parallel dumping with multiple
       threads, and file compression.

     * MySQL Shell’s new dump loading utility, loadDump(),
       supports the import of schemas dumped using MySQL Shell’s
       new instance dump utility and schema dump utility into a
       MySQL Database Service DB System. The dump loading
       utility provides data streaming from remote storage,
       parallel loading of tables or table chunks, progress
       state tracking, resume and reset capability, and the
       option of concurrent loading while the dump is taking
       place.

     * The X DevAPI implementation now supports JSON schema
       validation, which enables you to ensure that your
       documents have a certain structure before they can be
       inserted or updated in a collection. To enable or modify
       JSON schema validation you pass in a JSON object like:
{
    validation: {
      level: “off|strict”,
      schema: “json-schema”
    }
}

       Here, validation is JSON object which contains the keys
       you can use to configure JSON schema validation. The
       first key is level, which can take the value strict or
       off. The second key, schema, is a JSON schema, as defined
       at http://json-schema.org. If the level key is set to
       strict, documents are validated against the json-schema
       when they are added to the collection, or when an
       operation updates the document. If the document does not
       validate, the server generates an error and the operation
       fails. If the level key is set to off, documents are not
       validated against the json-schema.
       You can pass a validation JSON object to the
       schema.createCollection() operation, to enable JSON
       schema validation, and schema.modifyCollection()
       operation, to change the current JSON schema validation,
       for example to disable validation. For more information,
       see JSON Schema Validation
(https://dev.mysql.com/doc/x-devapi-userguide/en/collection-validation.html).

Bugs Fixed

     * MySQL Shell plugins now support the use of the **kwargs
       syntax in functions defined in Python that are made
       available by the plugin. Using **kwargs in a function
       definition lets you call the function using a
       variable-length list of keyword arguments with arbitrary
       names. If the function is called from MySQL Shell’s
       JavaScript mode, MySQL Shell passes the named arguments
       and their values into a dictionary object for the Python
       function. MySQL Shell first tries to associate a keyword
       argument passed to a function with any corresponding
       keyword parameter that the function defines, and if there
       is none, the keyword argument is automatically included
       in the **kwargs list. As a side effect of this support,
       any API function called from Python in MySQL Shell that
       has a dictionary of options as the last parameter
       supports defining these options using named arguments.
       (Bug #31495448)

     * When switching to SQL mode, MySQL Shell queries the SQL
       mode of the connected server to establish whether the
       ANSI_QUOTES mode is enabled. Previously, MySQL Shell
       could not proceed if it did not receive a result set in
       response to the query. The absence of a result is now
       handled appropriately. (Bug #31418783, Bug #99728)

     * In SQL mode, when the results of a query are to be
       printed in table format, MySQL Shell buffers the result
       set before printing, in order to identify the correct
       column widths for the table. With very large result sets,
       it was possible for this practice to cause an out of
       memory error. MySQL Shell now buffers a maximum of 1000
       rows for a result set before proceeding to format and
       print the table. Note that if a field in a row after the
       first 1000 rows contains a longer value than previously
       seen in that column in the result set, the table
       formatting will be misaligned for that row. (Bug
       #31304711)

     * Context switching in MySQL Shell’s SQL mode has been
       refactored and simplified to remove SQL syntax errors
       that could be returned when running script files using
       the source command. (Bug #31175790, Bug #31197312, Bug
       #99303)

     * The user account that is used to run MySQL Shell’s
       upgrade checker utility checkForServerUpgrade()
       previously required ALL privileges. The user account now
       requires only the RELOAD, PROCESS, and SELECT privileges.
       (Bug #31085098)

     * In Python mode, MySQL Shell did not handle invalid UTF-8
       sequences in strings returned by queries. (Bug #31083617)

     * MySQL Shell’s parallel table import utility importTable()
       has a new option characterSet, which specifies a
       character set encoding with which the input data file is
       interpreted during the import. Setting the option to
       binary means that no conversion is done during the
       import. When you omit this option, the import uses the
       character set specified by the character_set_database
       system variable to interpret the input data file. (Bug
       #31057707)

     * On Windows, if the MySQL Shell package was extracted to
       and used from a directory whose name contained multi-byte
       characters, MySQL Shell was unable to start. MySQL Shell
       now handles directory names with multi-byte characters
       correctly, including when setting up Python, loading
       prompt themes, and accessing credential helpers. (Bug
       #31056783)

     * MySQL Shell’s JSON import utility importJSON() now
       handles UTF-8 encoded files that include a BOM (byte mark
       order) at the start, which is the sequence 0xEF 0xBB
       0xBF. As a workaround in earlier releases, remove this
       byte sequence, which is not needed. (Bug #30993547, Bug
       #98836)

     * When the output format was set to JSON, MySQL Shell’s
       upgrade checker utility checkForServerUpgrade() included
       a description and documentation link for a check even if
       no issues were found. These are now omitted from the
       output, as they are with the text output format. (Bug
       #30950035)

On Behalf of Oracle/MySQL Release Engineering Team,
Sreedhar S

MySQL Shell 8.0.20 for MySQL Server 8.0 and 5.7 has been released

Dear MySQL users,

MySQL Shell 8.0.20 is a maintenance release of MySQL Shell 8.0 Series (a
component of the MySQL Server). The MySQL Shell is provided under
Oracle’s dual-license.

MySQL Shell 8.0 is highly recommended for use with MySQL Server 8.0 and
5.7. Please upgrade to MySQL Shell 8.0.20.

MySQL Shell is an interactive JavaScript, Python and SQL console
interface, supporting development and administration for the MySQL
Server. It provides APIs implemented in JavaScript and Python that
enable you to work with MySQL InnoDB cluster and use MySQL as a document
store.

The AdminAPI enables you to work with MySQL InnoDB cluster and InnoDB
ReplicaSet, providing integrated solutions for high availability and scalability
using InnoDB based MySQL databases, without requiring advanced MySQL
expertise.  For more information about how to configure and work with
MySQL InnoDB cluster and MySQL InnoDB ReplicaSet see

https://dev.mysql.com/doc/refman/en/mysql-innodb-cluster-userguide.html

The X DevAPI enables you to create “schema-less” JSON document
collections and perform Create, Update, Read, Delete (CRUD) operations
on those collections from your favorite scripting language.  For more
information about how to use MySQL Shell and the MySQL Document Store
support see

https://dev.mysql.com/doc/refman/en/document-store.html

For more information about the X DevAPI see

https://dev.mysql.com/doc/x-devapi-userguide/en/

If you want to write applications that use the the CRUD based X DevAPI
you can also use the latest MySQL Connectors for your language of
choice. For more information about Connectors see

https://dev.mysql.com/doc/index-connectors.html

For more information on the APIs provided with MySQL Shell see

https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/

and

https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/

Using MySQL Shell’s SQL mode you can communicate with servers using the
legacy MySQL protocol. Additionally, MySQL Shell provides partial
compatibility with the mysql client by supporting many of the same
command line options.

For full documentation on MySQL Server, MySQL Shell and related topics,
see

https://dev.mysql.com/doc/mysql-shell/8.0/en/

For more information about how to download MySQL Shell 8.0.20, see the
“General Availability (GA) Releases” tab at

http://dev.mysql.com/downloads/shell/

We welcome and appreciate your feedback and bug reports, see

http://bugs.mysql.com/

Enjoy and thanks for the support!

Changes in MySQL Shell 8.0.20 (2020-04-27, General Availability)

     * AdminAPI Added or Changed Functionality

     * AdminAPI Bugs Fixed

     * Functionality Added or Changed

     * Bugs Fixed

AdminAPI Added or Changed Functionality

     * MySQL Shell now enables you to create and configure the
       MySQL user accounts required by InnoDB cluster, InnoDB
       ReplicaSet, and MySQL Router using AdminAPI operations.
       Previously, accounts required by InnoDB cluster and
       InnoDB ReplicaSet had to be configured using the
       clusterAdmin option, and accounts required by MySQL
       Router had to be configured manually using SQL. The
       following AdminAPI operations are now available:

          + use Cluster.setupAdminAccount(user, [options]) and
            Replicaset.setupAdminAccount(user, [options]) to
            configure a MySQL user account with the necessary
            privileges to administer an InnoDB cluster or InnoDB
            ReplicaSet.

          + use Cluster.setupRouterAccount(user, [options]) and
            Replicaset.setupRouterAccount(user, [options]) to
            create a MySQL user account or upgrade an existing
            account so that it that can be used by MySQL Router
            to operate on an InnoDB cluster or InnoDB
            ReplicaSet. This is now the recommended method of
            adding MySQL Router accounts to use with InnoDB
            cluster and InnoDB ReplicaSet.

     * AdminAPI now uses a locking mechanism to avoid different
       operations from performing changes on an InnoDB
       ReplicaSet simultaneously. Previously, different
       instances of MySQL Shell could connect to an InnoDB
       ReplicaSet at the same time and execute AdminAPI
       operations simultaneously. This could lead to
       inconsistent instance states and errors, for example if
       ReplicaSet.addInstance() and ReplicaSet.setPrimary() were
       executed in parallel.
       Now, the InnoDB ReplicaSet operations have the following
       locking:

          + dba.upgradeMetadata() and dba.createReplicaSet() are
            globally exclusive operations. This means that if
            MySQL Shell executes these operations on an InnoDB
            ReplicaSet, no other operations can be executed
            against the InnoDB ReplicaSet or any of its
            instances.

          + ReplicaSet.forcePrimaryInstance() and
            ReplicaSet.setPrimaryInstance() are operations that
            change the primary. This means that if MySQL Shell
            executes these operations against an InnoDB
            ReplicaSet, no other operations which change the
            primary, or instance change operations can be
            executed until the first operation completes.

          + ReplicaSet.addInstance(),
            ReplicaSet.rejoinInstance(), and
            ReplicaSet.removeInstance() are operations that
            change an instance. This means that if MySQL Shell
            executes these operations on an instance, the
            instance is locked for any further instance change
            operations. However, this lock is only at the
            instance level and multiple instances in an InnoDB
            ReplicaSet can each execute one of this type of
            operation simultaneously. In other words, at most
            one instance change operation can be executed at a
            time, per instance in the InnoDB ReplicaSet.

          + dba.getReplicaSet() and ReplicaSet.status() are
            InnoDB ReplicaSet read operations and do not require
            any locking.
       References: See also: Bug #30349849.

     * Use the –replicaset option to configure MySQL Shell to
       work with an InnoDB ReplicaSet at start up. You must
       specify a connection to a replica set instance for this
       option to work correctly. If a replica set is found, this
       option populates the rs global object, which can then be
       used to work with the InnoDB ReplicaSet. As part of this
       addition, the –redirect-primary and –redirect-secondary
       options have been updated to also work with InnoDB
       ReplicaSet.
       When running MySQL Shell, use the
       shell.connectToPrimary([connectionData, password]) to
       check whether the target instance belongs to an InnoDB
       cluster or InnoDB ReplicaSet. If so, MySQL Shell opens a
       new session to the primary, sets the global session to
       the established session and returns it. If no
       connectionData is provided, the current global session is
       used.

AdminAPI Bugs Fixed

     * During distributed recovery which is using MySQL Clone,
       the instance restarts after the data files are cloned,
       but if the instance has to apply a large back log of
       transactions to finish the recovery process, then the
       restart process could take longer than the default 1
       minute timeout. Now, when there is a large back log use
       the dba.restartWaitTimeout option to configure a longer
       timeout to ensure the apply process has time to process
       the transactions. (Bug #30866632)

     * The dba.deleteSandboxInstance() operation did not provide
       an error if you attempted to delete a sandbox which did
       not exist. Now, in such a situation the
       dba.deleteSandboxInstance() operation throws a
       runtimeError. (Bug #30863587)

     * The Cluster.forceQuorumUsingPartitionOf() operation was
       not stopping Group Replication on any reachable instances
       that were not part of the visible membership of the
       target instance, which could lead to undefined behavior
       if any of those instances were automatically rejoining
       the cluster. The fix stops Group Replication on any
       reachable instances that are not included in the new
       forced quorum membership. (Bug #30739252)

     * It was possible for AdminAPI to select an invalidated
       instance as the latest primary, despite it having a lower
       view_id. This was because the process of getting the
       primary of the InnoDB ReplicaSet was incorrectly
       reconnecting to an invalidated member if it was the last
       instance in the InnoDB ReplicaSet. (Bug #30735124)

     * When a cluster that was created with a MySQL Shell
       version lower than 8.0.19 was offline (for example after
       a server upgrade of the instances), if you then used
       MySQL Shell 8.0.19 to connect to the cluster,
       dba.upgradeMetadata() and
       dba.rebootClusterFromCompleteOutage() blocked each other.
       You could not run dba.upgradeMetadata() because it
       requires dba.rebootClusterFromCompleteOutage() to be run
       first to bring the cluster back online. And you could not
       run dba.rebootClusterFromCompleteOutage() because
       dba.upgradeMetadata() had not been run. To avoid this
       problem please upgrade to MySQL Shell 8.0.20, where the
       preconditions for dba.rebootClusterFromCompleteOutage()
       and dba.forceQuorumUsingPartitionOf() have been updated
       to ensure they are compatible with clusters created using
       earlier versions. In other words, they are available even
       if the metadata was created using an older MySQL Shell
       version. (Bug #30661129)

     * Using MySQL Clone as the distributed recovery method to
       add an instance to an InnoDB ReplicaSet resulted in a
       segmentation fault if the target instance did not support
       RESTART. Now, the ReplicaSet.addInstance() operation
       aborts in such a situation and reverts changes after the
       connection timeout limit is reached. This is because the
       add operation needs to connect to the target instance to
       finish the operation. In such a situation, if it is not
       possible to upgrade the instance to a version of MySQL
       which supports RESTART, you have to restart the server
       manually, and then issue ReplicaSet.addInstance() again
       to retry. The retry can then use incremental recovery,
       which does not trigger the clone and the subsequent
       restart. (Bug #30657911)

     * Cluster.addInstance() normally fails if there are errant
       GTIDs in the added instance, but if MySQL Clone is being
       used for distributed recovery, that check is bypassed
       because the cloning process fixes the problem. However,
       if all members are using IPv6, MySQL Clone cannot be
       used, so incremental recovery is used instead. In such a
       situation, the instance was being added to the cluster
       without errors, but the errant transaction persisted.
       Now, if all of the cluster’s online instances are using
       IPv6 addresses and the operation tries to use MySQL Clone
       for distributed recovery, an error is thrown. (Bug
       #30645697)

     * When an InnoDB cluster or InnoDB ReplicaSet is using the
       MySQL Clone plugin, AdminAPI ensures the
       performance_schema.clone_status table is cleared out when
       the clone process starts. However, in some rare and very
       specific scenarios a race condition could happen and the
       clone operation was considered to be running before
       actually clearing out the table. In this situation, the
       MySQL Shell clone monitoring could result in an
       unexpected halt.
       As part of this fix, a potential infinite loop in the
       clone monitoring phase that could happen very rarely when
       the cloning process was extremely fast has also been
       fixed. (Bug #30645665)

     * Group Replication system variable queries were being
       executed early, without considering whether the Group
       Replication plugin was installed yet. Now, the reboot
       operation has been fixed so that if system variable
       queries fail with ER_UNKNOWN_SYSTEM_VARIABLE then the
       Group Replication plugin is installed automatically. (Bug
       #30531848)

     * The Cluster.removeInstance(instance) operation was not
       correctly handling the following cases:

          + if the instance had report_host set to a different
            value from the instance_name in the metadata,
            specifying the instance using its IP failed.

          + if the instance was unreachable, it was not possible
            to remove it if the given address did not match the
            address in the metadata.

          + when an instance was OFFLINE but reachable (for
            example because Group Replication stopped but the
            server was still running),
            Cluster.removeInstance(instance) failed. Now, in
            such a situation, if you are sure it is safe to
            remove the instance, use the force=true option,
            which means that synchronization is no longer
            attempted as part of the remove operation.

          + if the instance was OFFLINE but reachable, removing
            the instance through an address that did not match
            what was in the metadata would make the operation
            appear to succeed but the instance was not actually
            removed from the metadata.
       (Bug #30501628, Bug #30625424)

     * After operations such as removing an instance or
       dissolving a cluster, the group_replication_recovery and
       group_replication_applier replication channels were not
       being removed. (Bug #29922719, Bug #30878446)

     * The default location of the MySQL option file, for
       example /etc/my.cnf, stopped being detected by the
       dba.configureInstance() operation on some platforms
       (Debian and so on). This was a regression. The fix
       ensures that the predefined paths to option files matches
       the defaults, such as /etc/my.cnf and /etc/mysql/my.cnf.
       (Bug #96490, Bug #30171324)

Functionality Added or Changed

     * A new method shell.openSession is provided in the shell
       global object to let you create and return a session
       object, rather than set it as the global session for
       MySQL Shell.

     * You can now request compression for MySQL Shell
       connections that use X Protocol, as well as those that
       use classic MySQL protocol. For X Protocol connections,
       the default is that compression is requested, and
       uncompressed connections are allowed if the negotiations
       for a compressed connection do not succeed. For classic
       MySQL protocol connections, the default is that
       compression is disabled. After the connection has been
       made, the MySQL Shell \status command shows whether or
       not compression is in use for a session.
       New compression controls in MySQL Shell let you specify
       in the connection parameters whether compression is
       required, preferred, or disabled, select compression
       algorithms for the connection, and specify a numeric
       compression level for the algorithms.

Bugs Fixed

     * When you create an extension object for MySQL Shell, the
       options key is no longer required when you specify a
       parameter of the data type “dictionary”. If you do define
       options for a dictionary, MySQL Shell validates the
       options specified by the end user and raises an error if
       an option is passed to the function that is not in this
       list. If you create a dictionary with no list of options,
       any options that the end user specifies for the
       dictionary are passed directly through to the function by
       MySQL Shell with no validation. (Bug #30986260)

     * A bug in MySQL Shell 8.0.19, affecting classic MySQL
       protocol connections only, meant that access was denied
       if a user had stored the connection’s password with MySQL
       Shell and afterwards changed it. The password store now
       removes invalid passwords and presents the user with a
       password prompt as expected. (Bug #30912984, Bug #98503)

     * When MySQL Shell’s \source command was used in
       interactive mode to execute code from a script file,
       multi-line SQL statements in the script file could cause
       MySQL Shell to enter a loop of repeatedly executing the
       script. The issue has now been fixed. (Bug #30906751, Bug
       #98625)

     * If a stored procedure was called in MySQL Shell but its
       result was not used, any subsequent SQL statement
       returned a result set error, and exiting MySQL Shell at
       that point resulted in an incorrect shutdown. MySQL Shell
       cleared the first result set retrieved by a stored
       procedure in order to run a subsequent SQL statement, but
       did not check for any additional result sets that had
       been retrieved, which were left behind and caused the
       error. This check is now carried out and the additional
       result sets are discarded before another statement is
       executed. (Bug #30825330)

     * Due to a regression in MySQL Shell 8.0.19, the upgrade
       checker utility checkForServerUpgrade() did not accept
       any runtime options if connection data was not provided
       as the first argument. The issue has been fixed and the
       utility’s argument checking has been enhanced. (Bug
       #30689606)

     * MySQL Shell, which now bundles Python 3.7.4, could not be
       built from source with Python 3.8. The incompatibilities
       have now been corrected so Python 3.8 may be used. (Bug
       #30640012)

     * MySQL Shell’s upgrade checker utility
       checkForServerUpgrade() did not flag removed system
       variables that were specified using hyphens rather than
       underscores. The utility also now continues with its
       sequence of checks if a permissions check cannot be
       performed at the required time. (Bug #30615030, Bug
       #97855)

     * MySQL Shell’s \status command showed that a connection
       was compressed if the connection had been created while
       starting MySQL Shell, but not if it was created after
       starting MySQL Shell. Compression is now shown in both
       cases. (Bug #29006903)


On Behalf of Oracle/MySQL Release Engineering Team,
Sreedhar S

MySQL Shell 8.0.19 for MySQL Server 8.0 and 5.7 has been released

Dear MySQL users,

MySQL Shell 8.0.19 is a maintenance release of MySQL Shell 8.0 Series (a
component of the MySQL Server). The MySQL Shell is provided under
Oracle’s dual-license.

MySQL Shell 8.0 is highly recommended for use with MySQL Server 8.0 and
5.7. Please upgrade to MySQL Shell 8.0.19.

MySQL Shell is an interactive JavaScript, Python and SQL console
interface, supporting development and administration for the MySQL
Server. It provides APIs implemented in JavaScript and Python that
enable you to work with MySQL InnoDB cluster and use MySQL as a document
store.

The AdminAPI enables you to work with MySQL InnoDB cluster, providing an
integrated solution for high availability and scalability using InnoDB
based MySQL databases, without requiring advanced MySQL expertise. For
more information about how to configure and work with MySQL InnoDB
cluster see

https://dev.mysql.com/doc/refman/en/mysql-innodb-cluster-userguide.html

The X DevAPI enables you to create “schema-less” JSON document
collections and perform Create, Update, Read, Delete (CRUD) operations
on those collections from your favorite scripting language.  For more
information about how to use MySQL Shell and the MySQL Document Store
support see

https://dev.mysql.com/doc/refman/en/document-store.html

For more information about the X DevAPI see

https://dev.mysql.com/doc/x-devapi-userguide/en/

If you want to write applications that use the the CRUD based X DevAPI
you can also use the latest MySQL Connectors for your language of
choice. For more information about Connectors see

https://dev.mysql.com/doc/index-connectors.html

For more information on the APIs provided with MySQL Shell see

https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/

and

https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/

Using MySQL Shell’s SQL mode you can communicate with servers using the
legacy MySQL protocol. Additionally, MySQL Shell provides partial
compatibility with the mysql client by supporting many of the same
command line options.

For full documentation on MySQL Server, MySQL Shell and related topics,
see

https://dev.mysql.com/doc/mysql-shell/8.0/en/

For more information about how to download MySQL Shell 8.0.19, see the
“General Availability (GA) Releases” tab at

http://dev.mysql.com/downloads/shell/

We welcome and appreciate your feedback and bug reports, see

http://bugs.mysql.com/

Enjoy and thanks for the support!




Changes in MySQL Shell 8.0.19 (2020-01-13, General Availability)

InnoDB Cluster Added or Changed Functionality


     * Incompatible Change: The AdminAPI now includes InnoDB
       ReplicaSets, that enables you to administer asynchronous
       replication set ups in a similar way to InnoDB cluster.
       The addition of InnoDB ReplicaSet means that the InnoDB
       cluster metadata schema has been upgraded to version 2.0.
       Regardless of whether you plan to use InnoDB ReplicaSet
       or not, to use MySQL Shell 8.0.19 and AdminAPI you must
       upgrade the metadata of your clusters. Connect MySQL
       Shell’s global session to your cluster and use the new
       dba.upgradeMetadata() operation to upgrade the cluster’s
       metadata to use the new metadata.
       Warning
       Without upgrading the metadata you cannot use MySQL Shell
       8.0.19 to change the configuration of a cluster created
       with earlier versions. You can only read the
       configuration of the cluster, for example using
       Cluster.status().
       The dba.upgradeMetadata() operation upgrades any
       automatically created MySQL Router users to have the
       correct privileges. Manually created MySQL Router users
       with a name not starting with mysql_router_ are not
       automatically upgraded. This is an important step in
       upgrading your cluster, only then can the MySQL Router
       application be upgraded.
       Warning
       A cluster which is using the new metadata cannot be
       administered by earlier MySQL Shell versions, for example
       once you upgrade to version 8.0.19 you can no longer use
       version 8.0.18 or earlier to administer the cluster.
       To get information on which of the MySQL Router instances
       registered with a cluster require the metadata upgrade,
       issue
       cluster.listRouters({‘onlyUpgradeRequired’:’true’}).

     * AdminAPI now supports socket connections to be used for
       cluster and replica set operations. (Bug #26265826)

     * You can now get information about the MySQL Router
       instances registered with a InnoDB cluster, and
       unregister a Router from a cluster, for example when you
       stop using it. The list of Routers registered with a
       cluster contains information about whether the Router
       instance is compatible with the version of MySQL on the
       cluster instances, which guides you when upgrading a
       cluster. Use the Cluster.listRouters() operation to show
       a list of all Routers registered with the cluster. The
       returned JSON object provides information such as the
       hostname, ports, and if an upgrade is required. To filter
       the list to only show Router instances that do not
       support the latest metadata version use the
       onlyUpgradeRequired option, for example by issuing
       Cluster.listRouters({‘onlyUpgradeRequired’:’true’}). To
       remove a registered Router from a cluster, use the
       Cluster.removeRouterMetadata(router) operation.

     * The AdminAPI includes support for InnoDB ReplicaSet, that
       enables you to administer asynchronous replication sets
       in a similar way to InnoDB cluster. InnoDB ReplicaSet
       enables you to deploy an asynchronous replication set
       consisting of a single primary and multiple secondaries
       (traditionally referred to as the MySQL replication
       master and slaves). You administer a ReplicaSet using
       AdminAPI operations, for example to check the status of
       the InnoDB ReplicaSet, and manually failover to a new
       primary in the event of a failure. Similar to InnoDB
       cluster, MySQL Router supports bootstrapping against
       InnoDB ReplicaSet, which means you can automatically
       configure MySQL Router to use your InnoDB ReplicaSet
       without having to manually configure files. This makes
       InnoDB ReplicaSet a quick and easy way to get MySQL
       replication and MySQL Router up and running, making it
       well suited to scaling out reads, development
       environments, and applications that do not require the
       high availability offered by InnoDB cluster.

InnoDB Cluster Bugs Fixed


     * The dba.configureLocalInstance() operation could fail
       with a key not found (LogicError) error when executed on
       a non-sandbox instance where it did not have access to
       the my.cnf option file and the operation requested an
       output configuration file to be specified. (Bug
       #30657204)

     * If a cluster had been deployed with MySQL Shell version
       8.0.14 or earlier, the metadata contained an invalid port
       number for X Protocol connections. The metadata upgrade
       catches such ports and removes the invalid number. To
       avoid problems with routing due to this incorrect port,
       upgrade your cluster’s metadata. (Bug #30618264)

     * When a replication slave was configured to read from an
       InnoDB cluster primary, even with the appropriate
       replication filtering to ignore the metadata replication
       was failing when an instance was added to the cluster
       using MySQL Clone as the recovery method. This was
       because the recovery process was granting a privilege on
       an account, which failed and broke replication. (Bug
       #30609075)

     * The Cluster.removeInstance() operation was issuing a
       misleading error message when the instance was
       unreachable, indicating that it did not belong to the
       cluster when an alternative valid host or IP was used.
       Now, the error indicates that the instance is
       unreachable. (Bug #30580393)

     * Although MySQL Shell version 8.0.18 added support for
       IPv6 in WL#12758
       (https://dev.mysql.com/worklog/task/?id=12758), using an
       InnoDB cluster which consisted of MySQL Shell version
       8.0.18 and MySQL Router 8.0.18 with IPv6 addresses was
       not possible. With the release of version 8.0.19 of both
       MySQL Shell and MySQL Router, be aware that:

          + combining MySQL Shell version 8.0.18 with MySQL
            Router 8.0.18 causes Router to fail, due to no IPv6
            support. (BUG#30354273)

          + combining MySQL Shell version 8.0.18 with Router
            8.0.19 in a cluster which uses X Protocol
            connections, results in AdminAPI storing mysqlX IPv6
            values incorrectly in the metadata, causing Router
            to fail. (BUG#30548843) However, combining MySQL
            Shell version 8.0.18 with Router 8.0.19 in a cluster
            which uses MySQL classic protocol connections, is
            possible.
       Therefore, to use InnoDB cluster with IPv6 addresses,
       regardless of the protocol used, the recommended
       deployment is MySQL Shell 8.0.19 and MySQL Router 8.0.19.
       (Bug #30548843)
       References: See also: Bug #30354273.

     * When using automatic rejoin, if a target instance was
       rejoining the cluster, operations such as
       dba.rebootClusterFromCompleteOutage(), Cluster.status(),
       and so on were failing. Now, clusters consider automatic
       rejoin as an instance state instead of a check that
       always aborts the operation. This ensures that
       Cluster.status() is reported even for instances which are
       rejoining the cluster, and that
       dba.rebootClusterFromCompleteOutage() can detect
       instances which are rejoining the cluster and override
       the rejoin operation so that the cluster can be properly
       rebooted. (Bug #30501590)

     * SSL client certificate options for the clusterAdmin user
       were not being copied when setting up connection options,
       which made them fail when connecting. (Bug #30494198)

     * When the automatically calculated localAddress is not
       valid, for example when it exceeds the valid range, the
       error message has now been improved. See Configuring
       Ports
       (https://dev.mysql.com/doc/refman/8.0/en/mysql-innodb-cluster-production-deployment.html#configuring-instance-ports).
       (Bug #30405569)

     * The AdminAPI ensures that all members of a cluster have
       the same consistency level as configured at cluster
       creation time. However, when high and non-default
       consistency levels were chosen for the cluster, adding
       instances to it resulted in an error 3796 which indicates
       that group_replication_consistency cannot be used on the
       target instance. This happened because the consistency
       values of BEFORE, AFTER and BEFORE_AND_AFTER cannot be
       used on instances that are RECOVERING and several
       transactions happen while the instance is in the
       RECOVERING phase. Other AdminAPI commands result in the
       same error for the same scenario (high global consistency
       levels) whenever at least one member of the cluster is
       RECOVERING. For example, dba.getCluster(). The fix
       ensures that all sessions used by the AdminAPI use the
       consistency level of EVENTUAL when the cluster’s
       consistency level is BEFORE, AFTER or BEFORE_AND_AFTER.
       (Bug #30394258, Bug #30401048)

     * Some privileges required for persisting configuration
       changes on MySQL 8.0 servers were missing for the
       clusterAdmin users created by AdminAPI. In particular, an
       Access Denied error was being issued indicating the
       SYSTEM_VARIABLES_ADMIN and PERSIST_RO_VARIABLES_ADMIN
       privileges were required. Now these privileges are added
       for the clusterAdmin user on MySQL 8.0 servers. (Bug
       #30339460)

     * When using MySQL Clone as the recovery method, trying to
       add an instance that did not support RESTART to a cluster
       caused MySQL Shell to stop unexpectedly. Now, in such a
       situation a message explains that Cluster.rescan() must
       be used to ensure the instance is added to the metadata.
       (Bug #30281908)

     * The autocommit and sql_mode system variables are session
       settings, but they can be set globally to different
       values. AdminAPI was failing if these variables had
       non-default values in several different ways, for example
       DML was failing, system variables could not be set and so
       on. (Bug #30202883, Bug #30324461)

     * Attempting to bootstrap MySQL Router against an InnoDB
       cluster which had had the cluster administration user
       modified or removed was failing. This was caused by the
       privileges granted on the InnoDB cluster metadata table.
       The recommended solution is to upgrade to metadata 2.0,
       which changes the privileges on the metadata to ensure
       this issue does not occur. (Bug #29868432)

     * When you created a multi-primary cluster, the
       group_replication_enforce_update_everywhere_checks system
       variable was not being set automatically. However,
       switching to multi-primary mode automatically enables
       group_replication_enforce_update_everywhere_checks and
       switching to single-primary disables it. Now, the
       dba.createCluster() operation sets the
       group_replication_enforce_update_everywhere_checks
       variable as appropriate for single-primary or
       multi-primary clusters. (Bug #29794779)

     * In version 8.0.16, the autoRejoinTries option was added
       to define the number of times an instance attempts to
       rejoin the cluster after being expelled. The option is a
       valid cluster setting, configurable through the AdminAPI
       like many other options. However, the autoRejoinTries
       option was not being listed by Cluster.options(). (Bug
       #29654346)

     * The InnoDB cluster metadata now supports host names up to
       265 characters long, where 255 characters can be the host
       part and the remaining characters can be the port number.
       (Bug #29507913)

     * dba.createCluster() could fail if the instance had been
       started with innodb_default_row_format=COMPACT or
       innodb_default_row_format=REDUNDANT. This was because no
       ROW_FORMAT was specified on the InnoDB cluster metadata
       tables, which caused them to use the one defined in
       innodb_default_row_format. The metadata schema has been
       updated to use ROW_FORMAT = DYNAMIC. (Bug #28531271)

     * When an instance restarted, for example after a complete
       outage, it could have super_read_only disabled. This
       meant that instances which were not the primary could be
       written to, resulting in the instances no longer being in
       synchrony. This could result in
       dba.rebootClusterFromCompleteOutage() failing with a
       Conflicting transaction sets error. The fix ensures that
       all instances have super_read_only=1 persisted while they
       belong to the cluster, either through SET PERSIST_ONLY,
       or through dba.configureLocalInstance() for instances
       which do not support persisting. (Bug #97279, Bug
       #30545872)

     * The Cluster.status() operation could report an error
       get_uint(24): field value out of the allowed range
       because it was always expecting a positive value for some
       fields that could in fact have negative values. For
       example, this could happen when the clocks of different
       instances were offset. (Bug #95191, Bug #29705983)

     * If you changed the name of the clusterAdmin user once a
       cluster had been created, you could encounter an error
       such as The user specified as a definer does not exist.
       This was because the clusterAdmin user was used as the
       DEFINER of the views required by InnoDB cluster, and if
       this user is renamed then the definer is in effect
       missing. In version 8.0.19 the InnoDB cluster metadata
       has been changed to avoid this problem, use
       dba.upgradeMetadata() to upgrade the cluster. Clusters
       deployed with 8.0.19 and later do not suffer from this
       issue. (Bug #92128, Bug #28541069)

     * It was not possible to create a multi-primary cluster due
       to to cascading constraints on the InnoDB cluster
       metadata tables. This has been fixed in version 8.0.19
       and so to solve this issue upgrade your cluster using
       dba.upgradeMetadata(). (Bug #91972, Bug #29137199)

Functionality Added or Changed


     * The JavaScript function require() has been improved in
       MySQL Shell to support loading of local modules, in
       addition to built-in modules and modules that are on
       module search paths already known to MySQL Shell. If you
       specify the module name or path prefixed with ./ or ../,
       MySQL Shell now searches for the specified module in the
       folder that contains the JavaScript file or module
       currently being executed, or in interactive mode,
       searches in the current working directory. If the module
       is not found in that folder, MySQL Shell proceeds to
       check the well-known module search paths specified by the
       sys.path variable.

     * MySQL Shell’s upgrade checker utility (the
       util.checkForServerUpgrade() operation) includes the
       following new and extended checks:

          + The utility now flags all date, datetime, and
            timestamp columns that have a default value of zero,
            and states if the SQL mode (either global or for the
            current session) allows the insertion of zero values
            for these column types. By default, these are no
            longer permitted in MySQL, and it is strongly
            advised to replace zero values with valid ones, as
            they might not work correctly in the future.

          + The check for usage of removed functions now
            includes the PASSWORD() function.

          + The utility now checks for any orphaned tables which
            are recognized by InnoDB, but the SQL layer thinks
            they are handled by a different storage engine. This
            situation can happen if manual updates are made in
            the data directory. Orphaned tables can stall the
            upgrade process if they are present.

     * In MySQL Shell’s interactive mode, for JavaScript,
       Python, or SQL, the \source command or its alias \. can
       be used to execute code from a script file at a given
       path. For compatibility with the mysql client, in SQL
       mode only, you can now execute code from a script file
       using the source command with no backslash and an
       optional SQL delimiter. source can be used both in MySQL
       Shell’s interactive mode for SQL, to execute a script
       directly, and in a file of SQL code processed in batch
       mode, to execute a further script from within the file.
       In SQL mode only, you can also now use the \source
       command’s alias \. (which does not use a SQL delimiter)
       in a file of SQL code processed in batch mode. So with
       MySQL Shell in SQL mode, you could now execute the script
       in the /tmp/mydata.sql file from either interactive mode
       or batch mode using any of these three commands:
        source /tmp/mydata.sql;
        source /tmp/mydata.sql
        \. /tmp/mydata.sql

       The command \source /tmp/mydata.sql is also valid, but in
       interactive mode only.

Bugs Fixed


     * When searching for startup scripts in the platform’s
       standard global configuration path (in the folder
       %PROGRAMDATA%\MySQL\mysqlsh on Windows, or
/etc/mysql/mysqlsh/ on Unix), MySQL Shell checked for the
       incorrect script name shellrc, rather than the correct
       name mysqlshrc. (Bug #30656548)

     * On Windows, MySQL Shell passed UTF-8 encoded strings to
       the NTFS file system, which stores file names in UTF-16
       encoding. The mismatch caused files and folders to be
       incorrectly named or located when non-ASCII characters
       were used. MySQL Shell now converts all strings used in
       operations on NTFS from UTF-8 to UTF-16, and converts
       back to UTF-8 all strings received from Unicode function
       versions of Windows API calls. (Bug #30538516)

     * Some host names were not parsed correctly in the
       connection data provided when running MySQL Shell’s
       upgrade checker utility checkForServerUpgrade(). (Bug
       #30536355)

     * If the MySQL Server environment variable MYSQL_UNIX_PORT
       (which specifies the default Unix socket file) was
       updated by the same process that was then used to create
       a MySQL Shell connection to a MySQL server using a socket
       file, MySQL Shell cached and connected using the socket
       file path that had previously been set, but reported that
       a connection had been made using the updated socket file
       path. The correct socket file used for the connection is
       now displayed. (Bug #30533318)

     * If a prompt theme file used to customize the MySQL Shell
       prompt contained an ill-formed UTF-8 sequence, on startup
       an error message was displayed in place of the prompt
       text. MySQL Shell now validates the prompt theme file
       before loading it, and if there is a problem, uses a
       default prompt instead and issues an error message. (Bug
       #30406283)

     * If MySQL Shell was installed on Microsoft Windows at a
       non-default location, and subsequently uninstalled, files
       created after installation by the Python library used by
       MySQL Shell were not removed. These files are now removed
       when MySQL Shell is uninstalled from any location. (Bug
       #30333801)

     * Previously, most MySQL Shell options that expected an
       integer value could be set with an empty value, in which
       case the value 1 was applied. The exception was the
       logLevel option, which required a value. The behavior has
       now been standardized so all MySQL Shell options that
       expect a non-string value must be specified with a value,
       with the exception of options set on the command line.
       The affected options are dba.gtidWaitTimeout, dba.logSql,
       history.maxSize, and verbose. (Bug #30320839)

     * When using MySQL Shell in interactive mode, using a
       template literal in a multiple-line JavaScript statement
       resulted in an error. The issue has now been fixed. (Bug
       #30248651)

     * In Python mode, when multiple statements were input to
       MySQL Shell at the same time for execution in interactive
       mode, only the first statement was executed correctly.
       (Bug #30029568)

     * The Debian control file for MySQL Shell has been
       corrected to remove packaging errors that occurred when
       certain variables were not defined. Thanks to Evgeniy
       Patlan for the fix. (Bug #29802600, Bug #95158)

     * In MySQL Shell’s parser for URI-like connection strings,
       handling of path separators was previously platform
       dependent. Unified parsing has now been introduced so
       that Windows named pipes can be parsed correctly on Unix
       platforms, and Unix socket files can be parsed correctly
       on Windows platforms. (Bug #29456981)

     * MySQL Shell now looks up host names by obtaining the
       fully qualified domain name of the provided address and
       using the absolute form of this name (with a trailing
       dot). This method avoids potential issues caused by some
       network configurations that resolve host names as
       loopback addresses when they are actually addressable
       externally. (Bug #27704559)


On Behalf of Oracle/MySQL Release Engineering Team,
Hery Ramilison

MySQL Shell 8.0.18 for MySQL Server 8.0 and 5.7 has been released

Dear MySQL users,

MySQL Shell 8.0.18 is a maintenance release of MySQL Shell 8.0 Series (a
component of the MySQL Server). The MySQL Shell is provided under
Oracle’s dual-license.

MySQL Shell 8.0 is highly recommended for use with MySQL Server 8.0 and
5.7. Please upgrade to MySQL Shell 8.0.18.

MySQL Shell is an interactive JavaScript, Python and SQL console
interface, supporting development and administration for the MySQL
Server. It provides APIs implemented in JavaScript and Python that
enable you to work with MySQL InnoDB cluster and use MySQL as a document
store.

The AdminAPI enables you to work with MySQL InnoDB cluster, providing an
integrated solution for high availability and scalability using InnoDB
based MySQL databases, without requiring advanced MySQL expertise. For
more information about how to configure and work with MySQL InnoDB
cluster see

  https://dev.mysql.com/doc/refman/en/mysql-innodb-cluster-userguide.html

The X DevAPI enables you to create “schema-less” JSON document
collections and perform Create, Update, Read, Delete (CRUD) operations
on those collections from your favorite scripting language.  For more
information about how to use MySQL Shell and the MySQL Document Store
support see

  https://dev.mysql.com/doc/refman/en/document-store.html

For more information about the X DevAPI see

  https://dev.mysql.com/doc/x-devapi-userguide/en/

If you want to write applications that use the the CRUD based X DevAPI
you can also use the latest MySQL Connectors for your language of
choice. For more information about Connectors see

  https://dev.mysql.com/doc/index-connectors.html

For more information on the APIs provided with MySQL Shell see

  https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/

and

  https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/

Using MySQL Shell’s SQL mode you can communicate with servers using the
legacy MySQL protocol. Additionally, MySQL Shell provides partial
compatibility with the mysql client by supporting many of the same
command line options.

For full documentation on MySQL Server, MySQL Shell and related topics,
see

  https://dev.mysql.com/doc/mysql-shell/8.0/en/

For more information about how to download MySQL Shell 8.0.18, see the
“General Availability (GA) Releases” tab at

  http://dev.mysql.com/downloads/shell/

We welcome and appreciate your feedback and bug reports, see

  http://bugs.mysql.com/

Enjoy and thanks for the support!


Changes in MySQL Shell 8.0.18 (2019-10-14, General Availability)

     * InnoDB Cluster Added or Changed Functionality

     * InnoDB Cluster Bugs Fixed

     * Functionality Added or Changed

     * Bugs Fixed

InnoDB Cluster Added or Changed Functionality


     * MySQL Shell can now optionally log SQL statements that
       are executed by AdminAPI operations, and output them to
       the console if the –verbose option is set. The
       dba.logSql MySQL Shell configuration option or
       –dba-log-sql command line option activates logging for
       these statements. Statements executed by sandbox
       operations are excluded. Viewing the statements lets you
       observe the progress of the AdminAPI operations in terms
       of SQL execution, which can help with problem diagnosis
       for any errors.

     * AdminAPI now supports IPv6 addresses if the target MySQL
       Server version is higher than 8.0.13. When using MySQL
       Shell 8.0.18 or higher, if all cluster instances are
       running 8.0.14 or higher then you can use an IPv6 or
       hostname that resolves to an IPv6 address for instance
       connection strings and with options such as localAddress,
       groupSeeds and ipWhitelist. For more information on using
       IPv6 see Support For IPv6 And For Mixed IPv6 And IPv4
       Groups
(https://dev.mysql.com/doc/refman/8.0/en/group-replication-ipv6.html).
       References: See also: Bug #29557250, Bug #30111022, Bug
       #28982989.

     * You can now reset the passwords for the internal recovery
       accounts created by InnoDB cluster, for example to follow
       a custom password lifetime policy. Use the
       Cluster.resetRecoveryAccountsPassword() operation to
       reset the passwords for all internal recovery accounts
       used by the cluster. The operation sets a new random
       password for the internal recovery account on each
       instance which is ONLINE. If an instance cannot be
       reached, the operation fails. You can use the force
       option to ignore such instances, but this is not
       recommended, and it is safer to bring the instance back
       online before using this operation. This operation only
       applies to the passwords created by InnoDB cluster and
       cannot be used to update manually created passwords.
       Note
       The user which executes this operation must have all the
       required clusterAdmin privileges, in particular CREATE
       USER, in order to ensure that the password of recovery
       accounts can be changed regardless of the password
       verification-required policy. In other words, independent
       of whether the password_require_current system variable
       is enabled or not.

     * MySQL Shell now supports specifying TLS version 1.3 and
       TLS cipher suites for classic MySQL protocol connections.
       You can use:

          + the –tls-version command option to specify TLS
            version 1.3.

          + the –tls-ciphersuites command option to specify
            cipher suites.

          + the tls-versions and tls-ciphersuites connection
            parameters as part of a URI-type connection string.
       Note
       tls-versions (plural) does not have a key-value
       equivalent, it is only supported in URI-type connection
       strings. Use tls-version to specify TLSv1.3 in a
       key-value connection string.
       To use TLS version 1.3, both MySQL Shell and MySQL server
       must have been compiled with OpenSSL 1.1.1 or higher. For
       more information see Using Encrypted Connections
(https://dev.mysql.com/doc/refman/8.0/en/encrypted-connections.html).


InnoDB Cluster Bugs Fixed


     * The Cluster.rejoinInstance() operation was not setting
       the auto increment values defined for InnoDB cluster,
       leading to the use of the default Group Replication
       behavior if the instance configuration was not properly
       persisted, for example on 5.7 servers. The fix ensures
       that the Cluster.rejoinInstance() operation updates the
       auto increment settings of the target instance. (Bug
       #30174191)

     * The output of Cluster.status() now includes the
       replicationLag field. The value is displayed in HH:MM:SS
       format and shows the time difference between the last
       transaction commit timestamp and the last transaction
       applied timestamp. This enables you to monitor the amount
       of time between the most recent transaction being
       committed and being applied on an instance. (Bug
       #30003034)

     * Cluster.addInstance() did not ensure that the MySQL Clone
       plugin was installed or loaded on all cluster instances,
       when available and not disabled. This meant that whenever
       a cluster was created using an older MySQL Shell version,
       on a target MySQL instance supporting clone, the instance
       would not have the clone plugin installed. The result was
       that any Cluster.addInstance() call that used clone would
       fail. The same issue happened if an instance was added to
       a cluster consisting of one instance using the
       incremental recovery type and afterwards the seed
       instance was removed. This resulted in all cluster
       instances not having the clone plugin installed and
       therefore any instance added using the clone recovery
       method would fail. The fix ensures that the clone plugin
       is installed on all cluster members (if available and not
       disabled) at cluster creation time and also whenever an
       instance is added to a cluster. (Bug #29954085)

     * The Cluster.rejoinInstance() operation was not checking
       the GTID consistency of an instance being rejoined to a
       cluster, which could result in data diverging. Now, the
       GTID consistency checks conducted as part of the
       Cluster.rejoinInstance() operation have been improved to
       check for irrecoverable or diverged data-sets and also
       for empty GTID sets. If an instance is found to not be
       consistent with the cluster, it is not rejoined and the
       operation fails with a descriptive error. You are also
       shown the list of errant transactions, possible outcomes
       and solutions. (Bug #29953812)

     * Cluster.describe() was retrieving information about the
       cluster’s topology and the MySQL version installed on
       instances directly from the current session. Now, the
       information is retrieved from the Metadata schema, and
       the MySQL version is not included in the information
       output by Cluster.describe(). (Bug #29648806)

     * Using a password containing the ‘ character caused
       dba.deploySandbox() to fail. Now, all sensitive data is
       correctly wrapped to avoid such issues. (Bug #29637581)

     * The Cluster.addInstance() operation creates internal
       recovery users which are required by the Group
       Replication recovery process. If the
       Cluster.addInstance() operation failed, for example
       because Group Replication could not start, the created
       recovery users were not removed. Now, in the event of a
       failure any internal users are removed. (Bug #25503159)

     * When a cluster had lost quorum and the majority of the
       cluster instances were offline except the primary, after
       reestablishing quorum and adding a new instance to the
       cluster, it was not possible to remove and add the
       previous primary instance to the cluster. This was
       because the operation failed when trying to contact
       offline instances, which was because the feature to
       verify if a Group Replication protocol upgrade is
       required was not considering the possibility of some
       cluster instances being offline (not reachable). The fix
       improves the Group Replication protocol upgrade handling
       for the Cluster.removeInstance() operation, which now
       attempts to connect to other cluster instances and use
       the first reachable instance for this purpose. (Bug
       #25267603)

     * The dba.configureInstance() operation was not setting the
       binlog_checksum option with the required value (NONE) in
       the option file for instances that did not support SET
       PERSIST (for example instances running MySQL 5.7), when
       the option file path was not provided as an input
       parameter but instead specified though the operation
       wizard in interactive mode. (Bug #96489, Bug #30171090)

Functionality Added or Changed


     * MySQL Shell’s upgrade checker utility (the
       util.checkForServerUpgrade() operation) includes the
       following new and extended checks:

          + The utility now checks for tablespace names
            containing the string “FTS”, which can be
            incorrectly identified as tablespaces of full-text
            index tables, preventing upgrade. The issue has been
            fixed in MySQL 8.0.18, but affects upgrades to
            earlier MySQL 8.0 releases.

          + The check for database objects with names that
            conflict with reserved keywords now covers the
            additional keywords ARRAY, MEMBER, and LATERAL.

          + – The checks for obsolete sql_mode flags now check
            the global sql_mode setting.
       Running the upgrade checker utility no longer alters the
       gtid_executed value, meaning that the utility can be used
       on Group Replication group members without affecting
       their synchronization with the group. The upgrade checker
       also now works correctly with the ANSI_QUOTES SQL mode.
       (Bug #30002732, Bug #30103683, Bug #96351)
       References: See also: Bug #29992589.

     * MySQL Shell has two new built-in reports, which provide
       information drawn from various sources including MySQL’s
       Performance Schema:

          + threads lists the current threads in the connected
            MySQL server which belong to the user account that
            is used to run the report. Using the report-specific
            options, you can choose to show foreground threads,
            background threads, or all threads. You can report a
            default set of information for each thread, or
            select specific information to include in the report
            from a larger number of available choices. You can
            filter, sort, and limit the output.

          + thread provides detailed information about a
            specific thread in the connected MySQL server. By
            default, the report shows information on the thread
            used by the current connection, or you can identify
            a thread by its ID or by the connection ID. You can
            select one or more categories of information, or
            view all of the available information about the
            thread.
       You can run the new reports using MySQL Shell’s \show and
       \watch commands. The reports work with servers running
       all supported MySQL 5.7 and MySQL 8.0 versions. If any
       item of information is not available in the MySQL Server
       version of the target server, the reports leave it out.

     * MySQL Shell has two new control commands:

          + The \edit (\e) command opens a command in the
            default system editor for editing. If you specify an
            argument to the command, this text is placed in the
            editor, and if you do not, the last command in the
            MySQL Shell history is placed in the editor. When
            you have finished editing, MySQL Shell presents your
            edited text ready for you to execute or cancel. The
            command can also be invoked using the short form \e
            or the key combination Ctrl-X Ctrl-E.

          + The \system (\!) command runs the operating system
            command that you specify as an argument to the
            command, then displays the output from the command
            in MySQL Shell. MySQL Shell returns an error if it
            was unable to execute the command.

     * MySQL Shell now uses Python 3. For platforms that include
       a system supported installation of Python 3, MySQL Shell
       uses the most recent version available, with a minimum
       supported version of Python 3.4.3. For platforms where
       Python 3 is not included, MySQL Shell bundles Python
       3.7.4. MySQL Shell maintains code compatibility with
       Python 2.6 and Python 2.7, so if you require one of these
       older versions, you can build MySQL Shell from source
       using the appropriate Python version.

Bugs Fixed


     * In debug mode, MySQL Shell raised an assertion when
       handling a character contained in SQL strings. (Bug
       #30286680)

     * If a Python lambda was added as a member of a MySQL Shell
       extension object, the Python object was not released
       correctly when MySQL Shell shut down, causing a
       segmentation fault. (Bug #30156304)

     * A memory leak could occur when Python code was executed
       in interactive mode. (Bug #30138755)

     * Help information for a MySQL Shell report could not be
       displayed unless there was an active session. MySQL Shell
       now checks for an open session only before actually
       running the report. (Bug #30083371)

     * If a default schema was set for the MySQL Shell
       connection, and a different default schema was set after
       the connection was made, MySQL Shell’s \reconnect command
       attempted to use the default schema from the original
       connection. The user’s current default schema is now used
       for the reconnection attempt. (Bug #30059354)

     * Due to a bug introduced by a change in MySQL Shell
       8.0.16, the MSI file that is used by Windows Installer to
       install MySQL Shell overwrote the Windows PATH
       environment variable with the path to the application
       binary (mysqlsh), removing any other paths present. The
       issue has now been fixed. (Bug #29972020, Bug #95432)

     * When the \reconnect command is used to attempt
       reconnection to a server, if the last active schema set
       by the user appears to be no longer available, MySQL
       Shell now attempts to connect with no schema set. (Bug
       #29954572)

     * In interactive mode, MySQL Shell now handles multiline
       comments beginning with a slash and asterisk (/*) and
       ending with an asterisk and slash (*/). (Bug #29938424)

     * The MySQL Shell \source command was not handled correctly
       when used in combination with SQL statements. (Bug
       #29889926)

     * With MySQL Shell in SQL mode, if multiple SQL statements
       including a USE statement were issued on a single line
       with delimiters, the USE statement was not handled
       correctly. (Bug #29881615)

     * If MySQL Shell’s JSON import utility was used to send a
       large number of JSON documents to a server with
       insufficient processing capacity, the utility could fill
       up the write queue with batches of prepared documents,
       causing the connection to time out and the import to
       fail. The utility now waits to read the response from the
       server before sending the next batch of prepared
       documents to the server. (Bug #29878964)

     * When MySQL Shell was built from source with a bundled
       OpenSSL package, the required linker flags were not set.
       The issue has now been fixed. (Bug #29862189)

     * If a new query was executed in MySQL Shell while a result
       was still active, resulting in rows being cached, not all
       rows were returned by the old query. (Bug #29818714)


On Behalf of Oracle/MySQL Release Engineering Team,
Balasubramanian Kandasamy

MySQL Shell 8.0.16 for MySQL Server 8.0 and 5.7 has been released

Dear MySQL users,

MySQL Shell 8.0.16 is a maintenance release of MySQL Shell 8.0 Series (a
component of the MySQL Server). The MySQL Shell is provided under
Oracle’s dual-license.

MySQL Shell 8.0 is highly recommended for use with MySQL Server 8.0 and
5.7. Please upgrade to MySQL Shell 8.0.16.

MySQL Shell is an interactive JavaScript, Python and SQL console
interface, supporting development and administration for the MySQL
Server. It provides APIs implemented in JavaScript and Python that
enable you to work with MySQL InnoDB cluster and use MySQL as a document
store.

The AdminAPI enables you to work with MySQL InnoDB cluster, providing an
integrated solution for high availability and scalability using InnoDB
based MySQL databases, without requiring advanced MySQL expertise. For
more information about how to configure and work with MySQL InnoDB
cluster see

https://dev.mysql.com/doc/refman/en/mysql-innodb-cluster-userguide.html

The X DevAPI enables you to create “schema-less” JSON document
collections and perform Create, Update, Read, Delete (CRUD) operations
on those collections from your favorite scripting language. For more
information about how to use MySQL Shell and the MySQL Document Store
support see

https://dev.mysql.com/doc/refman/en/document-store.html

For more information about the X DevAPI see

https://dev.mysql.com/doc/x-devapi-userguide/en/

If you want to write applications that use the the CRUD based X DevAPI
you can also use the latest MySQL Connectors for your language of
choice. For more information about Connectors see

https://dev.mysql.com/doc/index-connectors.html

For more information on the APIs provided with MySQL Shell see

https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/

and

https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/

Using MySQL Shell’s SQL mode you can communicate with servers using the
legacy MySQL protocol. Additionally, MySQL Shell provides partial
compatibility with the mysql client by supporting many of the same
command line options.

For full documentation on MySQL Server, MySQL Shell and related topics,
see

https://dev.mysql.com/doc/mysql-shell/8.0/en/

For more information about how to download MySQL Shell 8.0.16, see the
“Generally Available (GA) Releases” tab at

http://dev.mysql.com/downloads/shell/

We welcome and appreciate your feedback and bug reports, see

http://bugs.mysql.com/

Enjoy and thanks for the support!

Changes in MySQL Shell 8.0.16 (2019-04-25, General Availability)

* Functionality Added or Changed

* Bugs Fixed

Functionality Added or Changed

* Important Change: Attempting to connect to an X Protocol
port, 33060 by default, using the classic MySQL protocol resulted
in the following error: ERROR 2013 (HY000): Lost connection to
MySQL server at 'reading initial communication packet', system
error: 0 This was because of differences in X Protocol and
classic MySQL protocol clients expectations on how connections
were initialized. Now, in such a situation the generated error
message is ERROR 2007 (HY000): Protocol mismatch; server version
= 11, client version = 10. If you encounter this error then you
are probably trying to use the wrong port for the protocol your
client is using. As part of this improvement the
mysqlx_enable_hello_notice system variable has been added, which
controls messages sent to classic MySQL protocol clients that try
to connect over X Protocol. When enabled, clients which do not
support X Protocol that attempt to connect to the server X
Protocol port receive an error explaining they are using the
wrong protocol. Set mysqlx_enable_hello_notice to false to permit
clients which do not recognize the hello message to still
connect.

* MySQL Shell's upgrade checker utility can now check the
configuration file (my.cnf or my.ini) for the server instance.
The utility checks for any system variables that are defined in
the configuration file but have been removed in the target MySQL
Server release, and also for any system variables that are not
defined in the configuration file and will have a different
default value in the target MySQL Server release. For these
checks, when you invoke checkForServerUpgrade(), you must provide
the file path to the configuration file. If you omit the file
path and the upgrade checker utility needs to run a check that
requires the configuration file, that check fails with a message
informing you that you must specify the file path. (Bug
#27801824, Bug #29222179)

* MySQL InnoDB cluster automatically and transparently
manages the communication protocol versions of its members,
whenever the cluster topology is changed using AdminAPI
operations. An InnoDB cluster always uses the most recent
communication protocol version that is supported by all instances
that are part of the cluster or joining it.

+ When an instance is added to, removed from, or
rejoins the cluster, or a rescan or reboot operation
is carried out on the cluster, the communication
protocol version is automatically set to a version
supported by the instance that is now at the
earliest MySQL Server version.

+ When you carry out a rolling upgrade by removing
instances from the cluster, upgrading them, and
adding them back into the cluster, the communication
protocol version is automatically upgraded when the
last remaining instance at the old MySQL Server
version is removed from the cluster prior to its
upgrade.
To see the communication protocol version in use in an
InnoDB cluster, use the Cluster.status() function with
the 'extended' option enabled. The communication protocol
version is returned in the 'GRProtocolVersion' field,
provided that the cluster has quorum and no cluster
members are unreachable.

* MySQL Shell now has a framework and commands that you can
use to set up and run reports to display live information from a
MySQL server, such as status and performance information. Reports
can be run once using the MySQL Shell \show command, or run then
refreshed continuously in a MySQL Shell session using the \watch
command. They can also be accessed as API functions in the
shell.reports object. The reporting facility supports both
built-in reports and user-defined reports. User-defined reports
can be created in the supported scripting languages JavaScript
and Python, and can be run in any MySQL Shell mode (JavaScript,
Python, or SQL), regardless of the language that the report was
written in. Reports can be saved in a folder in the MySQL Shell
configuration path and automatically loaded at startup. You can
also create a report directly in the MySQL Shell prompt. You
register a report to MySQL Shell using the shell.registerReport
method to provide information about the report and the options
and arguments that it supports. For more information, see
Reporting with MySQL Shell
(http://dev.mysql.com/doc/mysql-shell/8.0/en/mysql-shell-reporting.html).

* When running MySQL Shell in interactive mode, you can now
execute an SQL statement without switching to SQL mode and back
again afterwards. This function enables you to conveniently issue
some SQL statements in the context of a longer AdminAPI workflow
in JavaScript or Python mode. Use the \sql command immediately
followed by the SQL statement, for example:
\sql select * from sakila.actor limit 3;
The SQL statement does not need any additional quoting, and the
statement delimiter is optional. With this format, MySQL Shell
does not switch mode as it would if you entered the \sql command.
After the SQL statement has been executed, MySQL Shell remains in
JavaScript or Python mode.
You cannot use multiple line mode when you use the \sql command
with a query to execute single SQL statements while another
language is active. The command only accepts a single SQL query
on a single line.

* MySQL Shell history is now split per active language
which the command was issued under. This means that your history
now matches the active language, for example when you are running
in JavaScript mode having issued \js, the history contains the
previous JavaScript statements you issued, and when you issue
\sql to change to SQL mode your history contains the previous SQL
statements you issued. Similarly, now any history related
commands such as \history clear or \history delete are performed
on the history of the current active language. When you install
this version, any existing MySQL Shell history files are
duplicated to ensure that existing history is not lost.
Subsequent operations are then added to the language specific
history file.

* The new autoRejoinTries option enables you to configure
how many times an instance tries to rejoin a group after being
expelled. In scenarios where network glitches happen but recover
quickly, setting this option prevents you from having to manually
add the expelled instance back to the group. The autoRejoinTries
option accepts positive integer values between 0 and 2016 and the
default value is 0, which means that instances do not try to
automatically rejoin. Set the value to a valid integer to
configure the number of attempts expelled instances should make
to rejoin the group. You can pass the autoRejoinTries option to
these AdminAPI operations:

+ dba.createCluster()

+ Cluster.addInstance()

+ Cluster.setOption()

+ Cluster.setInstanceOption()
When you configure the autoRejoinTries option, it sets
the group_replication_autorejoin_tries system variable.
Passing the option to dba.createCluster(),
Cluster.addInstance() or Cluster.setInstanceOption()
configures the automatic rejoin for specific cluster
instances. Passing the option to Cluster.setOption()
configures the automatic rejoin for all cluster
instances.
For more information, see Responses to Failure Detection
and Network Partitioning
(http://dev.mysql.com/doc/refman/8.0/en/group-replication-responses-failure.html).

* When resultFormat was set to json or json/raw, every
result was being returned as a JSON document. This behavior was
expected when JSON wrapping is off (in other words the --json
command option was not used when starting MySQL Shell). Now, for
consistency reasons when JSON wrapping is off and resultFormat is
set to json or json/raw, every record is printed in a separate
document and statistics and warnings are printed in plain text.
For example if MySQL Shell is started without --json and
resultFormat=json/raw:
mysqlsh-sql> SHOW DATABASES;
{"Database":"information_schema"}
{"Database":"mysql"}
{"Database":"performance_schema"}
{"Database":"sys"}
4 rows in set (0.0035 sec)

If MySQL Shell is started with --json and with
resultFormat=json/raw:
mysqlsh-sql> SHOW DATABASES;
{
"hasData": true,
"rows": [
{
"Database": "information_schema"
},
{
"Database": "mysql"
},
{
"Database": "performance_schema"
},
{
"Database": "sys"
}
],
"executionTime": "0.0018 sec",
"affectedRowCount": 0,
"affectedItemsCount": 0,
"warningCount": 0,
"warningsCount": 0,
"warnings": [],
"info": "",
"autoIncrementValue": 0
}

* AdminAPI now reports information about the version of
MySQL running on instances. This information is available from
the following operations:

+ Cluster.status()

+ Cluster.describe()

+ Cluster.rescan()

See Checking the MySQL Version on Instances
(http://dev.mysql.com/doc/refman/8.0/en/mysql-innodb-cluster-working-
with-cluster.html#checking-version-on-instances) for
more information.

Bugs Fixed

* Removing an instance from a cluster when the instance to
be removed had no user defined for the group_replication_recovery
channel resulted in dropping users on the remaining instances of
the cluster. (Bug #29617572)

* MySQL Shell could be installed in an environment where
Python was not present, but the application has a dependency on
many standard Python modules, resulting in error messages at
startup. The RPM and Debian packages for MySQL Shell now
explicitly specify the dependency on Python. (Bug #29469201)

* The MSI file that is used by Windows Installer to install
MySQL Shell now adds the path to the application binary (mysqlsh)
to the Windows PATH environment variable, so that the application
can be started from a command prompt. (Bug #29457639)

* In the instructions to build MySQL Shell from source (the
INSTALL document), the required version of the optional V8
dependency has been updated from 3.28.71.19 to 6.7.288.46. Thanks
to Evgeniy Patlan for spotting this. (Bug #29430049, Bug #94529)

* The failoverConsistency option has been deprecated and a
new option named consistency has been added, to make it more
consistent with the target Group Replication
group_replication_consistency system variable name. The MySQL
Shell online documentation now also correctly describes all of
the values you can assign to the consistency option. (Bug
#29356599)

* The dba.configureLocalInstance() operation would remove
any section that did not start with mysqld from the provided
option file. This could remove sections such as the client
section from the option file. (Bug #29349014)

* MySQL Shell's upgrade checker utility
checkForServerUpgrade() could incorrectly report a schema
inconsistency error for a table whose name included a special
character such as a hyphen. (Bug #29346836, Bug #94303)

* When an instance with X Plugin disabled was added to an
InnoDB cluster, if the instance was later removed from the
cluster using Cluster.removeInstance() the operation failed with
LogicError "get_string(7): field is NULL". This was a regression
introduced by the fix for Bug#27677227. (Bug #29304183)

* There was an inconsistency between the behavior of
dba.checkInstanceConfiguration() and the commands to add
instances to the cluster (dba.createCluster() and
Cluster.addInstance()) regarding the localhost and loopback
address validation. In particular, a simple error was printed by
dba.checkInstanceConfiguration() but the execution of the
operation continued showing that everything was correct at the
end of the operation, while an error was issued and the execution
stopped for dba.createCluster() and Cluster.addInstance(). As
part of fixing this issue, it was decided that the existing
localhost and loopback address validations are no longer needed
and should be removed. In particular, whatever address is
specified for report_host, even if it is localhost or the
loopback address (127.0.0.1), should be allowed, because it was
explicitly specified by the user to use it. (Bug #29279941)

* The dba.rebootClusterFromCompleteOutage() operation was
not preserving the existing Group Replication configurations
previously set for the instances. In particular, the Group
Replication local address and exit state action values were being
changed. Now all settings are read at the time of rebooting the
cluster. (Bug #29265869)

* On Windows, MySQL Shell's upgrade checker utility
checkForServerUpgrade() incorrectly reported a schema
inconsistency error for partitioned tables. (Bug #29256562)

* Using either Cluster.setOption() or
Cluster.setInstanceOption() to set an option which only exists in
MySQL 8.0 on an instance running MySQL 5.7 was not being caught
correctly. (Bug #29246657)

* On Debian-based platforms (such as Ubuntu), if the
hostname resolved to 127.0.1.1 - which is the default on these
platforms - it was not possible to create a cluster using the
default settings. Now, in such situations a proper validation of
the instance is performed before creating a cluster and adding
instances to it. (Bug #29246110)

* MySQL Shell stopped unexpectedly if Python code was
running in interactive mode and threw exceptions from C++
libraries. These exceptions are now caught and translated to
Python's built-in RuntimeError exceptions. (Bug #29057116)

* The dba.checkInstanceConfiguration() operation did not
validate host restrictions for the account provided for cluster
administration, for example if the account could actually connect
to all of the instances in the cluster. In particular, now an
error is issued if the provided user account is only able to
connect through localhost. (Bug #29018457)

* When a connection is specified using key-value pairs in
MySQL Shell's shell.connect() method, the host name cannot be an
empty string. MySQL Shell now handles this situation consistently
and returns an error if the supplied host name is an empty
string. (Bug #28899522)

* InnoDB cluster configured auto_increment_increment and
auto_increment_offset on instances for clusters running in
multi-primary mode and consisting of up to 7 instances based on
the logic described at InnoDB cluster and Auto-increment
(http://dev.mysql.com/doc/refman/8.0/en/mysql-innodb-cluster-working-with-
cluster.html#mysql-innodb-cluster-auto-increment). But Group
Replication permits groups to contain up to 9 members, and
Cluster.addInstance() and Cluster.removeInstance() were not
following the logic used for other operations. Now, InnoDB
cluster uses the same logic for auto increment regardless of the
operation used and correctly handles multi-primary clusters with
more than 7 instances. (Bug #28812763)

* MySQL Shell's JSON import utility can now accept input
from FIFO special files (named pipes) when you invoke the utility
using the util.importJSON function, so you can carry out large
imports by this method without needing to put the data into a
file. (Bug #28785527)

* When you use the MySQL Shell command \help (or \h, or \?)
with a search pattern to search for help on a specific subject,
multiple help topic titles can match the pattern and be returned
as a list, to be selected by entering the command again with an
extended search pattern. With this system, it was possible for
help topics with a single-word title to be inaccessible from such
a list because there was nothing further to add to the search
pattern. To avoid this situation, the handling of multiple
matches has now been improved. If a topic title is found that
matches the given search pattern exactly (case-sensitive in the
event of multiple topic matches, and case-insensitive in the
event of no case-sensitive matches), the topic is identified as
the exact match and its help data is printed. The rest of the
topics with pattern matches in their titles are listed in a "see
also" section and can be selected by further pattern matching.
(Bug #28393119)

* MySQL Shell uses the host value of the provided
connection parameters as the target hostname used for AdminAPI
operations, namely to register the instance in the metadata (for
the dba.createCluster() and cluster.addInstance() operations).
However, the host used for the connection parameters might not
match the hostname that is used or reported by Group Replication,
which uses the value of the report_host system variable when it
is defined (in other words it is not NULL), otherwise the value
of hostname is used. Therefore, AdminAPI now follows the same
logic to register the target instance in the metadata and as the
default value for the group_replication_local_address variable on
instances, instead of using the host value from the instance
connection parameters. During this fix it was detected that when
the report_host variable was set to empty, Group Replication uses
an empty value for the host but AdminAPI (for example in commands
such as dba.checkInstanceConfiguration(),
dba.configureInstance(), dba.createCluster()) reports the
hostname as the value used which is inconsistent with the value
reported by Group Replication. An error is now issued by AdminAPI
if an empty value is set for the report_host system variable.
(Bug #28285389)

* In the event that dba.createCluster() failed and a
rollback was performed to remove the created replication
(recovery) users, the account created at localhost and any of the
ipWhitelist addresses were not being removed. The fix ensures
that the replication accounts are removed whenever a rollback
related to dba.createCluster() is performed. This work was based
on a code contribution from Bin Hong. (Bug #94182, Bug #29308037)

 

On Behalf of Oracle/MySQL Release Engineering Team,
Nawaz Nazeer Ahamed