I am experiencing a strange bug with generated columns and MariaDB running in a Docker container.
The image I'm using is mariadb:10.
I have been trying to add generated columns. The first column I add works fine; the second I add crashes the container and destroys the table.
Here's the first column that is working:
ALTER TABLE program
ADD is_current tinyint AS (
IF (
status IN ('active', 'suspended')
AND start_date >= NOW(),
1,
0
)
);
This one works just fine. The following SQL crashes the container:
ALTER TABLE program
ADD is_covered tinyint AS (
IF (
status IN ('active', 'suspended')
AND start_date <= NOW(),
1,
0
)
);
After restarting the container, I get the following errors:
SELECT * FROM program;
[42S02][1932] Table 'my_local.program' is marked as crashed and should be repaired
repair table my_local.program;
Table 'my_local.program' doesn't exist in engine / operation failed
Following the directions from this question I checked in the container for the existence of the ibdata1 file. It exists, as do the table's .ibd and .rfm files.
I have not been able to fix this; I had to drop the table and re-create it and re-import the data.
If anyone has any suggestions, I'd love to hear.
Checking the reference for MySQL 8 for generated columns I find that
Literals, deterministic built-in functions, and operators are
permitted. A function is deterministic if, given the same data in
tables, multiple invocations produce the same result, independently of
the connected user. Examples of functions that are nondeterministic
and fail this definition: CONNECTION_ID(), CURRENT_USER(), NOW().
This is also true of MySQL 5.7.
When I attempted to create your generated column with MySQL 8 I got this message:
Error Code: 3763. Expression of generated column 'is_covered' contains a disallowed function: now.
I note, however, that you are using mariadb:10. Although it is derived from MySQL, MariaDB is now effectively a different product.
The MariaDB reference for generated columns says: (for 10.2.1 onwards):
Non-deterministic built-in functions are supported in expressions for not indexed VIRTUAL generated columns.
Non-deterministic built-in functions are not supported in expressions for PERSISTENT or indexed VIRTUAL generated columns.
So, If you have MySQL you can't do this at all. If you have MariaDB 10.2.1+ you should be able to do it with certain limitations.
In any case, you should get an error message, not a crashed table. I suggest you check the MariaDB bug reports, and submit one if this is not already there.
Related
I mysteriously got an error in my Rails app locally:
PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "users_pkey"
DETAIL: Key (id)=(45) already exists.
The strange thing is that I didn't specify 45 as the ID. This number came from Postgres itself, which also then complained about it. I know this because when I tried it again I got the error with 46. The brute-force fix I used was to just repeat the insertion until it worked, therefore bringing Postgres' idea of the table's next available ID into line with reality.
500.times { User.create({employee_id: 1010101010101, blah_blah: "blah"}) rescue nil }
Since the employee_id has a unique constraint, any subsequent attempts to create the user after the first successful one would fail. And any previous to the first successful one would fail because Postgres tried to use an invalid id (primary key for the table).
So the brute-force approach works, but it's inelegant and it leaves me wondering what could have caused the database to get into this state. It also leaves me wondering how to check to see whether the production database is similarly inconsistent, and how to fix it (short of repeating the brute-force "fix").
Finding your Sequence
The first step to updating your sequence object is to figure out what the name of your sequence is. To find this, you can use the pg_get_serial_sequence() function.
SELECT pg_get_serial_sequence('table_name','id');
This will output something like public.person_id_seq, which is the relation name (regclass).
In Postgres 10+ there is also a pg_sequences view that you can use to find all sorts of information related to your sequences. The last_value column will show you the current value of the sequence:
SELECT * FROM pg_sequences;
Updating your Sequence
Once you have the sequence name, there are a few ways you can reset the sequence value:
1 - Use setval()
SELECT setval('public.person_id_seq',1020); -- Next value will be 1021
SELECT setval('public.person_id_seq',1020, False); -- Next value will be 1020
Source
2 - Use ALTER SEQUENCE (RESTART WITH)
ALTER SEQUENCE person_id_seq RESTART WITH 1030;
In this case, the value you provide (ex. 1030) will be the next value returned, so technically the sequence is being reset to <YOUR VALUE> - 1.
3 - Use ALTER SEQUENCE (START WITH, RESTART)
ALTER SEQUENCE person_id_seq START WITH 1030;
ALTER SEQUENCE person_id_seq RESTART;
Using this method is preferred if you need to repeatedly restart to a specific value. Subsequent calls to RESTART will reset the sequence to 1030 in this example.
This sort of thing happens when rows with specified IDs were inserted into the table. Since the IDs are specified, Postgres doesn't increment its sequence when inserting, and then the sequence becomes out of date with the data in the table. This could happen when manually inserted rows, or copying in rows from a CSV file, or replicating in rows, etc.
To avoid the issue, you simply need to let Postgres always handle the IDs, and never specify the ID yourself. However, if you've already messed up and need to fix the sequence, you can do so with the ALTER SEQUENCE command (using RESTART WITH or INCREMENT).
My last record's primary key was 552 & when I added a new record it's primary key allotted is 584.
I'm surprised & would like to know possible reasons for this behavior.
Application Details :
Server: Heroku hobby plan - dyno
Database: Heroku Postgres
Framework: Ruby on Rails
Additional Info -> I'm using rails admin panel to add new record
Possible reasons:
Some records were added+deleted
Inserting transaction was reverted for some reason, From postgres manual:
Note: Because smallserial, serial and bigserial are implemented using sequences, there may be "holes" or gaps in the sequence of values which appears in the column, even if no rows are ever deleted. A value allocated from the sequence is still "used up" even if a row containing that value is never successfully inserted into the table column. This may happen, for example, if the inserting transaction rolls back.
Corresponding sequence table_name_seq has increment more than 1 (probably not your case, sometimes is useful for sharding)
I am using Azure/Release to SQL Deploy. I have a DACPAC file to deploy, but I (1) have changed column datatypes, (2)deleted some columns on a table that has data already, (3) and added some FK columns. But because there is already production data on this, the deploy fails. What solution should be done on these scenarios?
SQL DEPLOY image
The column [dbo].[TableSample][ColumnSample] is being dropped, data loss could occur.
The type for column Description in table [dbo].[Table2] is currently NVARCHAR (1024) NULL but is being changed to NVARCHAR (100) NOT NULL. Data loss could occur.
The type for column Id in table [dbo].[Table3] is currently UNIQUEIDENTIFIER NOT NULL but is being changed to INT NOT NULL.
There is no implicit or explicit conversion. ***
The column [sampleColumnId] on table [dbo].[Table4] must be added
, but the column has no default value and does not allow NULL values. If the table contains data, the ALTER script will not work.
To avoid this issue you must either: add a default value to the column, mark it as allowing NULL values, or enable the generation of smart-defaults as a deployment option.
Your "data loss" warnings can likely be turned off by using the option to "allow data loss" in your publish options. It's just warning you that you are dropping columns or going to a smaller data length.
Your "Table3" change is just not going to work with keeping the data. GUIDs will not fit in an INT column. You might need to look at dropping/re-creating the table or renaming the current ID column to something else (OldId, maybe) and adding a new ID of type INT, probably with an Identity(1,1).
However the last column - you are trying to add a NOT NULL column with no default to an existing table. Either allow NULLs or put a named default value on the column so the column can be added.
I've recently taken over support of a system which uses Advantage Database Server as its back end. For some background, I have years of database experience but have never used ADS until now, so my question is purely about how to implement a standard pattern in this specific DBMS.
There's a stored procedure which has been previously developed which manages an ID column in this manner:
#ID = (SELECT ISNULL(MAX(ID), 0) FROM ExampleTable);
#ID = #ID + 1;
INSERT INTO Example_Table (ID, OtherStuff)
VALUES (#ID, 'Things');
--Do some other stuff.
UPDATE ExampleTable
SET AnotherColumn = 'FOO'
WHERE ID = #ID;
My problem is that I now need to run this stored procedure multiple times in parallel. As you can imagine, when I do this, the same ID value is getting grabbed multiple times.
What I need is a way to consistently create a unique value which I can be sure will be unique even if I run the stored procedure multiple times at the same moment. In SQL Server I could create an IDENTITY column called ID, and then do the following:
INSERT INTO ExampleTable (OtherStuff)
VALUES ('Things');
SET #ID = SCOPE_IDENTITY();
ADS has autoinc which seems similar, but I can't find anything conclusively telling me how to return the value of the newly created value in a way that I can be 100% sure will be correct under concurrent usage. The ADS Developer's Guide actually warns me against using autoinc, and the online help files offer functions which seem to retrieve the last generated autoinc ID (which isn't what I want - I want the one created by the previous statement, not the last one created across all sessions). The help files also list these functions with a caveat that they might not work correctly in situations involving concurrency.
How can I implement this in ADS? Should I use autoinc, some other built-in method that I'm unaware of, or do I genuinely need to do as the developer's guide suggests, and generate my unique identifiers before trying to insert into the table in the first place? If I should use autoinc, how can I obtain the value that has just been inserted into the table?
You use LastAutoInc(STATEMENT) with autoinc.
From the documentation (under Advantage SQL->Supported SQL Grammar->Supported Scalar Functions->Miscellaneous):
LASTAUTOINC(CONNECTION|STATEMENT)
Returns the last used autoinc value from an insert or append. Specifying CONNECTION will return the last used value for the entire connection. Specifying STATEMENT returns the last used value for only the current SQL statement. If no autoinc value has been updated yet, a NULL value is returned.
Note: Triggers that operate on tables with autoinc fields may affect the last autoinc value.
Note: SQL script triggers run on their own SQL statement. Therefore, calling LASTAUTOINC(STATEMENT) inside a SQL script trigger would return the lastautoinc value used by the trigger's SQL statement, not the original SQL statement which caused the trigger to fire. To obtain the last original SQL statement's lastautoinc value, use LASTAUTOINC(CONNECTION) instead.
Example: SELECT LASTAUTOINC(STATEMENT) FROM System.Iota
Another option is to use GUIDs.
(I wasn't sure but you may have already been alluding to this when you say "or do I genuinely need to do as the developer's guide suggests, and generate my unique identifiers before trying to insert into the table in the first place." - apologies if so, but still this info might be useful for others :) )
The use of GUIDs as a surrogate key allows either the application or the database to create a unique identifier, with a guarantee of no clashes.
Advantage 12 has built-in support for a GUID datatype:
GUID and 64-bit Integer Field Types
Advantage server and clients now support GUID and Long Integer (64-bit) data types in all table formats. The 64-bit integer type can be used to store integer values between -9,223,372,036,854,775,807 and 9,223,372,036,854,775,807 with no loss of precision. The GUID (Global Unique Identifier) field type is a 16-byte data structure. A new scalar function NewID() is available in the expression engine and SQL engine to generate new GUID. See ADT Field Types and Specifications and DBF Field Types and Specifications for more information.
http://scn.sap.com/docs/DOC-68484
For earlier versions, you could store the GUIDs as a char(36). (Think about your performance requirements here of course.) You will then need to do some conversion back and forth in your application layer between GUIDs and strings. If you're using some intermediary data access layer, e.g. NHibernate or Entity Framework, you should be able to at least localise the conversions to one place.
If some part of your logic is in a stored procedure, you should be able to use the newid() or newidstring() function, depending on the type of the backing column:
INSERT INTO Example_Table (newid(), OtherStuff)
In our Cassandra table, every time we change data-types of "collection-type" columns it start causing issue. For example:
For changing datatype from text to Map<text,float> we do this:
drop existing column
wait for cassandra to assimilate this change.
add column (same name) but different data-type.
This reflects fine in all nodes, but Cassandra logs start complaining during compaction with:
RuntimeException: 6d6...73 is not defined as a collection
I figured out the comparator entries are not correct in "system.schema_columnfamilies" table. Dropping table and recreating it fixes the problem but its not possible always.
Are there some best-practices when we are dealing with collection type columns in situations like above ?
database-version: DataStax-Enterprise: 4.7.1 Cassandra 2.1.8.621
cqlsh 5.0.1
I guess you stumbled upon one of those WAT moments in Cassandra. Its a bad practice to name a new column as previously dropped one. Sometimes it even doesn't work.
Regarding schema (or data) migrations take a look at our tool. It can help you execute schema updates while keeping data and populating fields.