Skip to content

Commit dcc642d

Browse files
committed
Add support for INSERT INTO table (...) SELECT ...
This feature has been in the works for a very long time, and has a lot of context... I've added headers so if you already know about the iteration of the API and the evolution of `InsertStatement` internally, skip to the third section. Getting to this API === I'd like to give a bit of context on the APIs that have been considered, and how I landed on this one. To preface, all of the iterations around this have been trying to carefully balance three things: - Easy to discover in the API - Being syntactically close to the generated SQL - Avoiding rightward drift For most of Diesel's life, our API was `insert(values).into(table)`. That API was originally introduced in 0.2 "to mirror `update` and `delete` (it didn't mirror them. It was backwards. It's always been backwards). My main concern with the old API actually was related to this feature. I couldn't come up with a decent API that had you specify the column list (you basically always need to specify the column list for this feature). So in 0.99 we changed it to what we have now, and I had toyed around with `insert_into(table).columns(columns).values(select)`, as well as `insert_into(table).from_select(columns, select)`. I was leaning towards the second one for a while (I didn't realize at the time that it was exactly SQLAlchemy's API). I hated the `columns` method because it was unclear what it was doing until you saw you were inserting from a select statement. It also implied an interaction with tuples that didn't exist. However, another thing that happened in 0.99 was that we deprecated our old upsert API. The `from_select` form reminded me far too much of the old `on_conflict` API, which we had just removed. In practice what that API would give you was something like this: ```rust insert_into(posts::table) .from_select( (posts::user_id, posts::title), users::table .select(( users::id, users::name.concat("'s First Post"), )), ) ``` That's just far too much rightward drift for me. Yes, you can assign the args to local variables, but they're awkward to name and now your code reads weird. I thought moving the columns list to another method call would help, but it doesn't. ```rust insert_into(posts::table) .from_select( users::table .select(( users::id, users::name.concat("'s First Post"), )), ) .into_columns((posts::user_id, posts::title)) ``` Eventually a member of the Diesel team had the idea of making this an `insert_into` method on select statements. This solves all of my concerns around rightward drift, though at the cost of the other two priorities. The API now looked like this: ``` users::table .select(( users::id, users::name.concat("'s First Post"), )) .insert_into(posts::table) .into_columns((posts::user_id, posts::title)) ``` I liked the way the code flowed, but I had concerns around discoverability, and didn't like how backwards it was from SQL. But I could live with it and started implementing. Up until this point I had been assuming that we would have an `InsertFromSelectStatement` struct, which was distinct from `InsertStatement` and necessitated the differently named methods. I realized when I started digging into it though, that we really just want to re-use `InsertStatement` for this. It seems obvious in hindsight. And if we were going to use that structure, that meant that it wouldn't be much harder to just make passing `SelectStatement` to `values` work. This automatically solves most of my concerns around discoverability, since it now just works exactly like every other form of insert. That said, I really don't like the rightward drift. I liked the `.insert_into` form for being able to avoid that. But for the final implementation, I just generalized that feature. *Anything* that can be written as `insert_into(table).values(values)` can now be written as `values.insert_into(table)`. Context around InsertStatement === This file has churned more than just about any other part of Diesel. I feel like I've re-written it nearly every release at this point. I think the reason it's churned so much is for two reasons. The first is that it's kept a fundamental design flaw through 1.1 (which I'll get to), and we've been constantly working around it. The second is that `INSERT` is actually one of the most complex queries in SQL. It has less variations than `SELECT`, but far more of its variations are backend specific, or have different syntaxes between backends. `InsertStatement` was originally added for 0.2 in c9894b3 which has a very helpful commit message "WIP" (good job past Sean). At the time we only supported PG, so we actually had two versions -- `InsertStatement` and `InsertQuery`, the latter having a returning clause. I'm honestly not sure why I didn't do the `ReturningClause<T>` `NoReturningClause` dance we do now and did back then in `SelectStatement`. Maybe I thought it'd be easier? Anyway this file had to go through some pretty major changes in 0.5 when we added SQLite support. We needed to disallow batch insert and returning clauses on that backend. Additionally, the way we handle default values had to work differently on SQLite since it doesn't support the `DEFAULT` keyword. At this point though, it still a few things. The query starts with `INSERT INTO`, had a columns list, then the word `VALUES` and then some values. It also managed all parenthesis. (Yes it was trivial to make it generate invalid SQL at that point). Fast forward to 0.9, I decided to support batch insert on SQLite. This needs to do one query per row, and I felt that meant we needed a separate struct. I didn't want to have `BatchInsertStatement` and `BatchInsertQuery` and `InsertStatement` and `InsertQuery`, so we got the `NoReturningClause` struct, and got a little closer to how literally every other part of Diesel works. In 0.14 we added `insert_default_values` which left us with `InsertStatement`, `BatchInsertStatement`, and `DefaultInsertStatement`, which eventually went back down to the two. The last time the file went through a big rewrite was in 0.99 when I finally unified it down to the one struct we have today. However, it still had the fatal flaw I mentioned earlier. It was trying to handle too much logic, and was too disjoint from the rest of the query builder. It assumed that the two forms were `DEFAULT VALUES` or `(columns) VALUES ...`, and also handled parens. We've gone through a lot of refactoring to get rid of that. I finally think this struct is at a point where it will stop churning, mostly because it looks like the rest of Diesel now. It doesn't do anything at all, and the only thing it assumes is that the word `INTO` appears in the query (which I'm pretty sure actually is true). The actual implementation of this commit ==== With all that said, this commit is relatively simple. The main addition is the `InsertFromSelect` struct. It's definitely a Diesel struct, in that it basically does nothing, and all the logic is in its `where` clauses. I tried to make `Insertable` work roughly everywhere that methods like `filter` work. I couldn't actually do a blanket impl for tables in Rust itself, because it made rustc vomit on recursion with our impls on `Option`. That shouldn't be happening, but I suspect it's a lot of work to fix that in the language. I've also implemented it for references, since most of the time that you pass values to `.values`, you pass a reference. That should "just work" here unless we have a good reason not to. The majority of the additions here are tests. This is a feature that fundamentally interacts with many of our most complex features, and I've tried to be exhaustive. Theres ~3 lines of tests for every line of code added. I've done at least a minimal test of this feature's interaction with every other feature on inserts that I could think of. I am not so much worried about whether they work, but I was worried about if there was a syntactic edge case I didn't know about. There weren't. Same with the compile tests, I've tried to think of every way the feature could be accidentally misused (bad arguments to `into_columns` basically), as well as all the nonsense things we should make sure don't work (putting it in a tuple or vec). I didn't add any compile tests on making sure that the select statement itself is valid. The fact that the `Query` bound only matches valid complete select statements is already very well tested, and we don't need to re-test that here. I also did not explicitly disallow selecting from the same table as the insert, as this appears to be one of the few places where the table can appear twice with no ambiguity. Fixes diesel-rs#1106
1 parent c866ae9 commit dcc642d

16 files changed

Lines changed: 757 additions & 18 deletions

File tree

CHANGELOG.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,15 @@ for Rust libraries in [RFC #1105](https://github.com/rust-lang/rfcs/blob/master/
88

99
### Added
1010

11+
* Added support for `INSERT INTO table (...) SELECT ...` queries. Tables, select
12+
select statements, and boxed select statements can now be used just like any
13+
other `Insertable` value.
14+
15+
* Any insert query written as `insert_into(table).values(values)` can now be
16+
written as `values.insert_into(table)`. This is particularly useful when
17+
inserting from a select statement, as select statements tend to span multiple
18+
lines.
19+
1120
* Added support for specifying `ISOLATION LEVEL`, `DEFERRABLE`, and `READ ONLY`
1221
on PG transactions. See [`PgConnection::build_transaction`] for details.
1322

diesel/src/insertable.rs

Lines changed: 65 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ use std::marker::PhantomData;
33
use backend::{Backend, SupportsDefaultKeyword};
44
use expression::{AppearsOnTable, Expression};
55
use result::QueryResult;
6-
use query_builder::{AstPass, QueryFragment, UndecoratedInsertRecord, ValuesClause};
6+
use query_builder::{AstPass, InsertStatement, QueryFragment, UndecoratedInsertRecord, ValuesClause};
77
use query_source::{Column, Table};
88
#[cfg(feature = "sqlite")]
99
use sqlite::Sqlite;
@@ -20,21 +20,79 @@ use sqlite::Sqlite;
2020
/// struct differs from the name of the column, you can annotate the field
2121
/// with `#[column_name = "some_column_name"]`.
2222
pub trait Insertable<T> {
23+
/// The `VALUES` clause to insert these records
24+
///
25+
/// The types used here are generally internal to Diesel.
26+
/// Implementations of this trait should use the `Values`
27+
/// type of other `Insertable` types.
28+
/// For example `<diesel::dsl::Eq<column, &str> as Insertable<table>>::Values`.
2329
type Values;
2430

31+
/// Construct `Self::Values`
32+
///
33+
/// Implementations of this trait typically call `.values`
34+
/// on other `Insertable` types.
2535
fn values(self) -> Self::Values;
36+
37+
/// Insert `self` into a given table.
38+
///
39+
/// `foo.insert_into(table)` is identical to `insert_into(table).values(foo)`.
40+
/// However, when inserting from a select statement,
41+
/// this form is generally preferred.
42+
///
43+
/// # Example
44+
///
45+
/// ```rust
46+
/// # #[macro_use] extern crate diesel;
47+
/// # include!("doctest_setup.rs");
48+
/// #
49+
/// # fn main() {
50+
/// # run_test().unwrap();
51+
/// # }
52+
/// #
53+
/// # fn run_test() -> QueryResult<()> {
54+
/// # use schema::{posts, users};
55+
/// # let conn = establish_connection();
56+
/// # diesel::delete(posts::table).execute(&conn)?;
57+
/// users::table
58+
/// .select((
59+
/// users::name.concat("'s First Post"),
60+
/// users::id,
61+
/// ))
62+
/// .insert_into(posts::table)
63+
/// .into_columns((posts::title, posts::user_id))
64+
/// .execute(&conn)?;
65+
///
66+
/// let inserted_posts = posts::table
67+
/// .select(posts::title)
68+
/// .load::<String>(&conn)?;
69+
/// let expected = vec!["Sean's First Post", "Tess's First Post"];
70+
/// assert_eq!(expected, inserted_posts);
71+
/// # Ok(())
72+
/// # }
73+
/// ```
74+
fn insert_into(self, table: T) -> InsertStatement<T, Self::Values>
75+
where
76+
Self: Sized,
77+
{
78+
::insert_into(table).values(self)
79+
}
2680
}
2781

2882
pub trait CanInsertInSingleQuery<DB: Backend> {
29-
fn rows_to_insert(&self) -> usize;
83+
/// How many rows will this query insert?
84+
///
85+
/// This function should only return `None` when the query is valid on all
86+
/// backends, regardless of how many rows get inserted.
87+
fn rows_to_insert(&self) -> Option<usize>;
3088
}
3189

3290
impl<'a, T, DB> CanInsertInSingleQuery<DB> for &'a T
3391
where
3492
T: ?Sized + CanInsertInSingleQuery<DB>,
3593
DB: Backend,
3694
{
37-
fn rows_to_insert(&self) -> usize {
95+
fn rows_to_insert(&self) -> Option<usize> {
3896
(*self).rows_to_insert()
3997
}
4098
}
@@ -43,17 +101,17 @@ impl<'a, T, Tab, DB> CanInsertInSingleQuery<DB> for BatchInsert<'a, T, Tab>
43101
where
44102
DB: Backend + SupportsDefaultKeyword,
45103
{
46-
fn rows_to_insert(&self) -> usize {
47-
self.records.len()
104+
fn rows_to_insert(&self) -> Option<usize> {
105+
Some(self.records.len())
48106
}
49107
}
50108

51109
impl<T, U, DB> CanInsertInSingleQuery<DB> for ColumnInsertValue<T, U>
52110
where
53111
DB: Backend,
54112
{
55-
fn rows_to_insert(&self) -> usize {
56-
1
113+
fn rows_to_insert(&self) -> Option<usize> {
114+
Some(1)
57115
}
58116
}
59117

diesel/src/macros/mod.rs

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -707,6 +707,7 @@ macro_rules! table_body {
707707
JoinTo,
708708
};
709709
use $crate::associations::HasTable;
710+
use $crate::insertable::Insertable;
710711
use $crate::query_builder::*;
711712
use $crate::query_builder::nodes::Identifier;
712713
use $crate::query_source::{AppearsInFromClause, Once, Never};
@@ -861,6 +862,30 @@ macro_rules! table_body {
861862
}
862863
}
863864

865+
// This impl should be able to live in Diesel,
866+
// but Rust tries to recurse for no reason
867+
impl<T> Insertable<T> for table
868+
where
869+
<table as AsQuery>::Query: Insertable<T>,
870+
{
871+
type Values = <<table as AsQuery>::Query as Insertable<T>>::Values;
872+
873+
fn values(self) -> Self::Values {
874+
self.as_query().values()
875+
}
876+
}
877+
878+
impl<'a, T> Insertable<T> for &'a table
879+
where
880+
table: Insertable<T>,
881+
{
882+
type Values = <table as Insertable<T>>::Values;
883+
884+
fn values(self) -> Self::Values {
885+
(*self).values()
886+
}
887+
}
888+
864889
/// Contains all of the columns of this table
865890
pub mod columns {
866891
use super::table;

diesel/src/pg/upsert/on_conflict_clause.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ impl<Values, Target, Action> CanInsertInSingleQuery<Pg> for OnConflictValues<Val
3333
where
3434
Values: CanInsertInSingleQuery<Pg>,
3535
{
36-
fn rows_to_insert(&self) -> usize {
36+
fn rows_to_insert(&self) -> Option<usize> {
3737
self.values.rows_to_insert()
3838
}
3939
}

diesel/src/query_builder/functions.rs

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,47 @@ pub fn delete<T: IntoUpdateTarget>(source: T) -> DeleteStatement<T::Table, T::Wh
249249
/// # }
250250
/// ```
251251
///
252+
/// ### Insert from select
253+
///
254+
/// When inserting from a select statement,
255+
/// the column list can be specified with [`.into_columns`].
256+
/// (See also [`SelectStatement::insert_into`], which generally
257+
/// reads better for select statements)
258+
///
259+
/// [`SelectStatement::insert_into`]: prelude/trait.Insertable.html#method.insert_into
260+
/// [`.into_columns`]: query_builder/struct.InsertStatement.html#method.into_columns
261+
///
262+
/// ```rust
263+
/// # #[macro_use] extern crate diesel;
264+
/// # include!("../doctest_setup.rs");
265+
/// #
266+
/// # fn main() {
267+
/// # run_test().unwrap();
268+
/// # }
269+
/// #
270+
/// # fn run_test() -> QueryResult<()> {
271+
/// # use schema::{posts, users};
272+
/// # let conn = establish_connection();
273+
/// # diesel::delete(posts::table).execute(&conn)?;
274+
/// let new_posts = users::table
275+
/// .select((
276+
/// users::name.concat("'s First Post"),
277+
/// users::id,
278+
/// ));
279+
/// diesel::insert_into(posts::table)
280+
/// .values(new_posts)
281+
/// .into_columns((posts::title, posts::user_id))
282+
/// .execute(&conn)?;
283+
///
284+
/// let inserted_posts = posts::table
285+
/// .select(posts::title)
286+
/// .load::<String>(&conn)?;
287+
/// let expected = vec!["Sean's First Post", "Tess's First Post"];
288+
/// assert_eq!(expected, inserted_posts);
289+
/// # Ok(())
290+
/// # }
291+
/// ```
292+
///
252293
/// ### With return value
253294
///
254295
/// ```rust
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
use backend::Backend;
2+
use query_builder::*;
3+
use query_source::Column;
4+
use result::QueryResult;
5+
6+
/// Represents the column list for use in an insert statement.
7+
///
8+
/// This trait is implemented by columns and tuples of columns.
9+
pub trait ColumnList {
10+
/// The table these columns belong to
11+
type Table;
12+
13+
/// Generate the SQL for this column list.
14+
///
15+
/// Column names must *not* be qualified.
16+
fn walk_ast<DB: Backend>(&self, out: AstPass<DB>) -> QueryResult<()>;
17+
}
18+
19+
impl<C> ColumnList for C
20+
where
21+
C: Column,
22+
{
23+
type Table = <C as Column>::Table;
24+
25+
fn walk_ast<DB: Backend>(&self, mut out: AstPass<DB>) -> QueryResult<()> {
26+
out.push_identifier(C::NAME)?;
27+
Ok(())
28+
}
29+
}
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
use backend::Backend;
2+
use expression::{Expression, NonAggregate, SelectableExpression};
3+
use insertable::*;
4+
use query_builder::*;
5+
use query_source::Table;
6+
7+
/// Represents `(Columns) SELECT FROM ...` for use in an `INSERT` statement
8+
#[derive(Debug, Clone, Copy)]
9+
pub struct InsertFromSelect<Select, Columns> {
10+
query: Select,
11+
columns: Columns,
12+
}
13+
14+
impl<Select, Columns> InsertFromSelect<Select, Columns> {
15+
/// Construct a new `InsertFromSelect` where the target column list is
16+
/// `T::AllColumns`.
17+
pub fn new<T>(query: Select) -> Self
18+
where
19+
T: Table<AllColumns = Columns>,
20+
Columns: SelectableExpression<T> + NonAggregate,
21+
{
22+
Self {
23+
query,
24+
columns: T::all_columns(),
25+
}
26+
}
27+
28+
/// Replace the target column list
29+
pub fn with_columns<C>(self, columns: C) -> InsertFromSelect<Select, C> {
30+
InsertFromSelect {
31+
query: self.query,
32+
columns,
33+
}
34+
}
35+
}
36+
37+
impl<DB, Select, Columns> CanInsertInSingleQuery<DB> for InsertFromSelect<Select, Columns>
38+
where
39+
DB: Backend,
40+
{
41+
fn rows_to_insert(&self) -> Option<usize> {
42+
None
43+
}
44+
}
45+
46+
impl<DB, Select, Columns> QueryFragment<DB> for InsertFromSelect<Select, Columns>
47+
where
48+
DB: Backend,
49+
Columns: ColumnList + Expression<SqlType = Select::SqlType>,
50+
Select: Query + QueryFragment<DB>,
51+
{
52+
fn walk_ast(&self, mut out: AstPass<DB>) -> QueryResult<()> {
53+
out.push_sql("(");
54+
self.columns.walk_ast(out.reborrow())?;
55+
out.push_sql(") ");
56+
self.query.walk_ast(out.reborrow())?;
57+
Ok(())
58+
}
59+
}
60+
61+
impl<Select, Columns> UndecoratedInsertRecord<Columns::Table> for InsertFromSelect<Select, Columns>
62+
where
63+
Columns: ColumnList + Expression<SqlType = Select::SqlType>,
64+
Select: Query,
65+
{
66+
}

0 commit comments

Comments
 (0)