Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wip/PoC: Enrich metadata custom types with essential types for chain communications #4358

Closed
wants to merge 24 commits into from
Closed
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
f0ea720
metadata/ir: Extend IR with custom types
lexnv May 1, 2024
fecdd47
metadata/ir: Convert custom IR types to concrete metadata
lexnv May 1, 2024
1279594
frame/metadata: Populate custom types with runtime call of system config
lexnv May 1, 2024
ab4f6cc
frame/metadata: Expose AccountId to metadata
lexnv May 1, 2024
e7bee31
frame/metadata: Optionally expose the AssetId of the asset pallet
lexnv May 1, 2024
d0986cf
frame/metadata: Use runtime instance for extracting AssetId type
lexnv May 2, 2024
c8061aa
frame/metadata: Extract address ty
lexnv May 2, 2024
e63dae6
frame/metadata: Extract signature ty
lexnv May 2, 2024
c666a6f
frame/metadata: Extend with Hash and Hashing types
lexnv May 2, 2024
fbd75b0
frame/metadata: Extract block header to custom types
lexnv May 2, 2024
0fedb4d
frame/metadata: Remove call enum ty from custom types
lexnv May 2, 2024
17ade23
frame/proc: Refactor itemTrait extraction
lexnv May 16, 2024
616e38d
frame/proc: Refactor check_event_type
lexnv May 16, 2024
482a5ed
frame/proc: Refactor TraitItemType parsing
lexnv May 16, 2024
785436c
frame/constants: Improve documentation
lexnv May 17, 2024
a4d7ec7
frame/helpers: Refactor take_first_item_pallet_attr
lexnv May 17, 2024
c50d364
frame/config: Add associated types to parsed config
lexnv May 17, 2024
f40acd4
metadata-ir: Add type for PalletAssociatedTypesMetadataIR
lexnv May 17, 2024
7659b19
frame/config: Implement associated types metadata
lexnv May 17, 2024
ab0b1b1
frame/construct_runtime: Extract associated types from pallet config
lexnv May 17, 2024
b1d37e2
Add typeInfo to types and custom cargo patch
lexnv May 17, 2024
97396d0
frame/proc: Extract cfg attributes
lexnv May 17, 2024
3f5597e
frame/config: Add type attributes to metadata expansion
lexnv May 17, 2024
a6532ed
Add type info on other config types
lexnv May 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ pub fn expand_runtime_metadata(
scrate: &TokenStream,
extrinsic: &TokenStream,
system_path: &PalletPath,
assets_pallet: Option<&Pallet>,
header: &TokenStream,
) -> TokenStream {
let pallets = pallet_declarations
.iter()
Expand Down Expand Up @@ -75,6 +77,23 @@ pub fn expand_runtime_metadata(
})
.collect::<Vec<_>>();

let maybe_asset_id = assets_pallet
.map(|pallet| {
let path = &pallet.path;
let instance = &pallet.instance;
let maybe_instance = instance.as_ref().map(|inst| quote! { <#inst> });

quote! {
(
"AssetId".into(),
#scrate::__private::scale_info::meta_type::<
<#runtime as #path::Config #maybe_instance>::AssetId
>(),
),
}
})
.unwrap_or_default();

quote! {
impl #runtime {
fn metadata_ir() -> #scrate::__private::metadata_ir::MetadataIR {
Expand Down Expand Up @@ -140,6 +159,43 @@ pub fn expand_runtime_metadata(
>(),
event_enum_ty: #scrate::__private::scale_info::meta_type::<RuntimeEvent>(),
error_enum_ty: #scrate::__private::scale_info::meta_type::<RuntimeError>(),
},
custom_types: #scrate::__private::metadata_ir::CustomMetadataIR {
map: [
(
"AccountId".into(),
#scrate::__private::scale_info::meta_type::<
<#runtime as #system_path::Config>::AccountId
>(),
),
(
"Address".into(),
address_ty,
),
(
"Signature".into(),
signature_ty,
),
Comment on lines +173 to +180
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They already exist in the extrinsic metadata?

(
"Hash".into(),
#scrate::__private::scale_info::meta_type::<
<#runtime as #system_path::Config>::Hash
>(),
),
(
"Hashing".into(),
#scrate::__private::scale_info::meta_type::<
<#runtime as #system_path::Config>::Hashing
>(),
),
(
"Header".into(),
#scrate::__private::scale_info::meta_type::<#header>(),
),
#maybe_asset_id
]
.into_iter()
.collect(),
}
}
}
Expand Down
13 changes: 13 additions & 0 deletions substrate/frame/support/procedural/src/construct_runtime/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,10 @@ use syn::{spanned::Spanned, Ident, Result};

/// The fixed name of the system pallet.
const SYSTEM_PALLET_NAME: &str = "System";
/// The fixed name of the ForeignAssets pallet.
const FOREIGN_ASSETS_PALLET_NAME: &str = "ForeignAssets";
/// The fixed name of the Assets pallet.
const ASSETS_PALLET_NAME: &str = "Assets";
Comment on lines +229 to +232
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nothing we want to integrate. Users will have no fucking idea that they should call the pallets this way. These pallets can also have multiple instances. This doesn't makes any sense and we should not start doing this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep that makes sense! Its a bit of a hack to see how far I can get in subxt to remove those config types :D

I think a better approach may be to collect the associated types from pallet's configs. However, I'll have to look how much that will increase the metadata size and if we run into limitations with it (maybe not all types impl TypeInfo for us to expose these), will think about it! Thanks for the review! 🙏


/// Implementation of `construct_runtime` macro. Either expand to some code which will call
/// `construct_runtime` again, or expand to the final runtime definition.
Expand Down Expand Up @@ -365,6 +369,12 @@ fn construct_runtime_final_expansion(
))
}

// Find either the foreign assets pallet or the local assets pallet in this order.
let assets_pallet = pallets
.iter()
.find(|decl| decl.name == FOREIGN_ASSETS_PALLET_NAME)
.or_else(|| pallets.iter().find(|decl| decl.name == ASSETS_PALLET_NAME));

let features = pallets
.iter()
.filter_map(|decl| {
Expand All @@ -388,6 +398,7 @@ fn construct_runtime_final_expansion(
let frame_system = generate_access_from_frame_or_crate("frame-system")?;
let block = quote!(<#name as #frame_system::Config>::Block);
let unchecked_extrinsic = quote!(<#block as #scrate::sp_runtime::traits::Block>::Extrinsic);
let header = quote!(<#block as #scrate::sp_runtime::traits::Block>::Header);

let outer_event =
expand::expand_outer_enum(&name, &pallets, &scrate, expand::OuterEnumType::Event)?;
Expand All @@ -406,6 +417,8 @@ fn construct_runtime_final_expansion(
&scrate,
&unchecked_extrinsic,
&system_pallet.path,
assets_pallet,
&header,
);
let outer_config = expand::expand_outer_config(&name, &pallets, &scrate);
let inherent =
Expand Down
13 changes: 13 additions & 0 deletions substrate/frame/support/procedural/src/runtime/expand/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,10 @@ use syn::{Ident, Result};

/// The fixed name of the system pallet.
const SYSTEM_PALLET_NAME: &str = "System";
/// The fixed name of the ForeignAssets pallet.
const FOREIGN_ASSETS_PALLET_NAME: &str = "ForeignAssets";
/// The fixed name of the Assets pallet.
const ASSETS_PALLET_NAME: &str = "Assets";

pub fn expand(def: Def, legacy_ordering: bool) -> TokenStream2 {
let input = def.input;
Expand Down Expand Up @@ -143,6 +147,12 @@ fn construct_runtime_final_expansion(
))
}

// Find either the foreign assets pallet or the local assets pallet in this order.
let assets_pallet = pallets
.iter()
.find(|decl| decl.name == FOREIGN_ASSETS_PALLET_NAME)
.or_else(|| pallets.iter().find(|decl| decl.name == ASSETS_PALLET_NAME));

let features = pallets
.iter()
.filter_map(|decl| {
Expand All @@ -166,6 +176,7 @@ fn construct_runtime_final_expansion(
let frame_system = generate_access_from_frame_or_crate("frame-system")?;
let block = quote!(<#name as #frame_system::Config>::Block);
let unchecked_extrinsic = quote!(<#block as #scrate::sp_runtime::traits::Block>::Extrinsic);
let header = quote!(<#block as #scrate::sp_runtime::traits::Block>::Header);

let mut dispatch = None;
let mut outer_event = None;
Expand Down Expand Up @@ -230,6 +241,8 @@ fn construct_runtime_final_expansion(
&scrate,
&unchecked_extrinsic,
&system_pallet.path,
assets_pallet,
&header,
);
let outer_config = expand::expand_outer_config(&name, &pallets, &scrate);
let inherent =
Expand Down
25 changes: 24 additions & 1 deletion substrate/primitives/metadata-ir/src/types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
use codec::Encode;
use scale_info::{
form::{Form, MetaForm, PortableForm},
prelude::vec::Vec,
prelude::{collections::BTreeMap, vec::Vec},
IntoPortable, MetaType, Registry,
};

Expand All @@ -41,6 +41,8 @@ pub struct MetadataIR<T: Form = MetaForm> {
pub apis: Vec<RuntimeApiMetadataIR<T>>,
/// The outer enums types as found in the runtime.
pub outer_enums: OuterEnumsIR<T>,
/// The custom types collected by the runtime.
pub custom_types: CustomMetadataIR<T>,
}

/// Metadata of a runtime trait.
Expand Down Expand Up @@ -368,6 +370,27 @@ impl From<MetaType> for PalletEventMetadataIR {
}
}

/// Metadata for custom types.
#[derive(Clone, PartialEq, Eq, Encode, Debug)]
pub struct CustomMetadataIR<T: Form = MetaForm> {
/// The custom map.
pub map: BTreeMap<T::String, T::Type>,
}

impl IntoPortable for CustomMetadataIR {
type Output = CustomMetadataIR<PortableForm>;

fn into_portable(self, registry: &mut Registry) -> Self::Output {
let map = self
.map
.into_iter()
.map(|(key, value)| (key.into_portable(registry), registry.register_type(&value)))
.collect();

CustomMetadataIR { map }
}
}

/// Metadata about one pallet constant.
#[derive(Clone, PartialEq, Eq, Encode, Debug)]
pub struct PalletConstantMetadataIR<T: Form = MetaForm> {
Expand Down
28 changes: 19 additions & 9 deletions substrate/primitives/metadata-ir/src/v15.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,16 @@

//! Convert the IR to V15 metadata.

use crate::OuterEnumsIR;

use super::types::{
ExtrinsicMetadataIR, MetadataIR, PalletMetadataIR, RuntimeApiMetadataIR,
RuntimeApiMethodMetadataIR, RuntimeApiMethodParamMetadataIR, SignedExtensionMetadataIR,
CustomMetadataIR, ExtrinsicMetadataIR, MetadataIR, OuterEnumsIR, PalletMetadataIR,
RuntimeApiMetadataIR, RuntimeApiMethodMetadataIR, RuntimeApiMethodParamMetadataIR,
SignedExtensionMetadataIR,
};

use frame_metadata::v15::{
CustomMetadata, ExtrinsicMetadata, OuterEnums, PalletMetadata, RuntimeApiMetadata,
RuntimeApiMethodMetadata, RuntimeApiMethodParamMetadata, RuntimeMetadataV15,
SignedExtensionMetadata,
CustomMetadata, CustomValueMetadata, ExtrinsicMetadata, OuterEnums, PalletMetadata,
RuntimeApiMetadata, RuntimeApiMethodMetadata, RuntimeApiMethodParamMetadata,
RuntimeMetadataV15, SignedExtensionMetadata,
};

impl From<MetadataIR> for RuntimeMetadataV15 {
Expand All @@ -38,9 +37,8 @@ impl From<MetadataIR> for RuntimeMetadataV15 {
ir.ty,
ir.apis.into_iter().map(Into::into).collect(),
ir.outer_enums.into(),
// Substrate does not collect yet the custom metadata fields.
// This allows us to extend the V15 easily.
CustomMetadata { map: Default::default() },
ir.custom_types.into(),
)
}
}
Expand Down Expand Up @@ -119,3 +117,15 @@ impl From<OuterEnumsIR> for OuterEnums {
}
}
}

impl From<CustomMetadataIR> for CustomMetadata {
fn from(ir: CustomMetadataIR) -> Self {
CustomMetadata {
map: ir
.map
.into_iter()
.map(|(name, ty)| (name, CustomValueMetadata { ty, value: Default::default() }))
.collect(),
}
}
}
Loading