docs: add documentation describing API Notes

[compnerd]

Summary

API Notes are a feature which allows annotation of headers by an
auxiliary file that contains metadata for declarations pertaining to the
associated module. This enables adding attributes to declarations
without requiring modification of the headers, enabling finer grained
control for library headers for consumers without having to modify
external headers.

Based on the work from https://github.com/llvm/llvm-project-staging/commit/d238492fd3a82160955d0642b85d8ce620fa3258

[rsmith]

Broadly, it seems reasonable to me for Clang to support this. I have no major concerns with the overall approach here, and it seems like you already have sufficient implementation experience with this approach to know that it's going to work out well in practice. I'm happy that all of the points in http://clang.llvm.org/get_involved.html for Clang extensions are covered -- or will be covered by the work to implement this.

clang/docs/APINotes.rst

6–13

"API" should generally be used only as an uncountable noun. If you mean something more concrete here, such as "a function, method, type, ...", then saying that would be better, since "API" can be interpreted as meaning an individual such entity or a collection of such entities, depending on the reader.

46

Can we add a hyphen between api and notes here? -fapinotes is a little hard to read.

188–191

Is it important that these are single letters? Spelling out the name in full (as you do for other enumerated values like MethodKind and PropertyKind) would seem a little more readable. (We could accept the single-letter forms as aliases.)

234–236

So what context is it parsed in? Below you have an NSArray * example; how do we do the lookup for NSArray?

274

[compnerd]

clang/docs/APINotes.rst

46

Sure; we can always add a -fapinotes-modules as a silent alias for backwards compatibility. I assume you would like the same for the rest of the options, and have made that change through out.

188–191

I don't think that they be single letters is important. As long as we have the compatibility aliases, I think it should be fine to support the fully spelt out versions, as the compatibility is needed for existing APINotes.

234–236

A separate buffer is constructed where the annotations are processed. During semantic analysis, the requested APINotes are processed and the attributes specified are applied to the declaration.

[hlopko]

Thanks for pushing this forward!

Current proposal seems to be only dealing with annotating ObjC for Swift, but API notes seem like a feature that would be useful for other languages as well. I briefly chatted with gribozavr and we concluded that if the need arises, the current design should be extensible to e.g. C++ (one of the bigger problems being how to name C++ function/overload sets).

clang/docs/APINotes.rst

188–191

+1, this is the only attribute that is abbreviated.

217

Could we get more context from the Apple bug tracker? Depending on the details it might be reasonable to drop NullabilityOfRet attribute completely, or more precisely document when it should be used.

[compnerd]

clang/docs/APINotes.rst

217

The bug is that Nullability and Type on Parameters entries don’t really compose. Consider the following note:

NullabilityOfRet: N
Parameters:
- Position: 1
  Nullability: O
  Type: 'NSDictionary<NSString *, id> *'

on a method: this parameter won’t be treated as optional. Effectively, you need to write the _Nullable on the type because the Nullability entry is ignored.

I think that fixing this later is preferable rather than earlier to ensure that we at least have a unified starting point where clang and Swift can co-evolve. If we try to do that while upstreaming, it will make everything far more difficult because you have to coordinate over multiple (3+) repositories.

[compnerd]

Update text based on feedback.

[rsmith]

clang/docs/APINotes.rst

234–236

I've not been able to work out what the rule is, based on this documentation. The name NSArray isn't predeclared, so how is it found? How can a user ensure their types are visible to this parsing process? Are these things parsed as if they appear at the global scope in a translation unit in which the corresponding framework has been imported? (And if so, why wouldn't macros be available there too?) Whatever the rule is, I'd like to see it documented.

[compnerd]

clang/docs/APINotes.rst

234–236

Discussed offline, this has been updated.

[compnerd]

Address feedback from @rsmith

[rsmith]

Thanks, looks good to me.

clang/docs/APINotes.rst

251–253

Please update this text similarly.

clang/docs/APINotes.rst

+================================================
+API Notes: Annotations Without Modifying Headers
+================================================
+
+**The Problem:** You have headers you want to use, but you also want to add
+extra information to the API. You don't want to put that information in the
+headers themselves --- perhaps because you want to keep them clean for other
+clients, or perhaps because they're from some open source project and you don't
+want to modify them at all.
+
+**Incomplete solution:** Redeclare all the interesting parts of the API in your
+own header and add the attributes you want. Unfortunately, this:
+
+* doesn't work with attributes that must be present on a definition
+* doesn't allow changing the definition in other ways
+* requires your header to be included in any client code to take effect
+
+**Better solution:** Provide a "sidecar" file with the information you want to
+add, and have that automatically get picked up by the module-building logic in
+the compiler.
+
+That's API notes.
+
+API notes use a YAML-based file format. YAML is a format best explained by
+example, so here is a `small example`__ from the compiler test suite of API
+notes for a hypothetical "SomeKit" framework.
+
+__ test/APINotes/Inputs/Frameworks/SomeKit.framework/Headers/SomeKit.apinotes
+
+
+Usage
+=====
+
+API notes files are found relative to the module map that defines a module,
+under the name "SomeKit.apinotes" for a module named "SomeKit". Additionally, a
+file named "SomeKit_private.apinotes" will also be picked up to go with a
+private module map. For bare modules these two files will be in the same
+directory as the corresponding module map; for framework modules, they should
+be placed in the Headers and PrivateHeaders directories, respectively. The
+module map for a private top-level framework module should be placed in the
+PrivateHeaders directory as well, though it does not need an additional
+"_private" suffix on its name.
+
+Clang will search for API notes files next to module maps only when passed the
+``-fapi-notes-modules`` option.
+
+
+Limitations
+===========
+
+- Since they're identified by module name, API notes cannot be used to modify
+  arbitrary textual headers.
+
+
+"Versioned" API Notes
+=====================
+
+Many API notes affect how a C API is imported into Swift. In order to change
+that behavior while still remaining backwards-compatible, API notes can be
+selectively applied based on the Swift compatibility version provided to the
+compiler (e.g. ``-fapi-notes-swift-version=5``). The rule is that an
+explicitly-versioned API note applies to that version *and all earlier
+versions,* and any applicable explicitly-versioned API note takes precedence
+over an unversioned API note.
+
+
+Reference
+=========
+
+An API notes file contains a YAML dictionary with the following top-level
+entries:
+
+:Name:
+
+  The name of the module (the framework name, for frameworks). Note that this
+  is always the name of a top-level module, even within a private API notes
+  file.
+
+  ::
+
+    Name: MyFramework
+
+:Classes, Protocols, Tags, Typedefs, Globals, Enumerators, Functions:
+
+  Arrays of top-level declarations. Each entry in the array must have a
+  'Name' key with its Objective-C name. "Tags" refers to structs, enums, and
+  unions; "Enumerators" refers to enum cases.
+
+  ::
+
+    Classes:
+    - Name: MyController
+      …
+    - Name: MyView
+      …
+
+:SwiftVersions:
+
+  Contains explicit information for backwards compatibility. Each entry in
+  the array contains a 'Version' key, which should be set to '4' for
+  annotations that only apply to Swift 4 mode and earlier. The other entries
+  in this dictionary are the same declaration entries as at the top level:
+  Classes, Protocols, Tags, Typedefs, Globals, Enumerators, and Functions.
+
+  ::
+
+    SwiftVersions:
+    - Version: 4
+      Classes: …
+      Protocols: …
+
+Each entry under 'Classes' and 'Protocols' can contain "Methods" and
+"Properties" arrays, in addition to the attributes described below:
+
+:Methods:
+
+  Identified by 'Selector' and 'MethodKind'; the MethodKind is either
+  "Instance" or "Class".
+
+  ::
+
+    Classes:
+    - Name: UIViewController
+      Methods:
+      - Selector: "presentViewController:animated:"
+        MethodKind: Instance
+        …
+
+:Properties:
+
+  Identified by 'Name' and 'PropertyKind'; the PropertyKind is also either
+  "Instance" or "Class".
+
+  ::
+
+    Classes:
+    - Name: UIView
+      Properties:
+      - Name: subviews
+        PropertyKind: Instance
+        …
+
+Each declaration supports the following annotations (if relevant to that
+declaration kind), all of which are optional:
+
+:SwiftName:
+
+  Equivalent to ``NS_SWIFT_NAME``. For a method, must include the full Swift name
+  with all arguments. Use "_" to omit an argument label.
+
+  ::
+
+    - Selector: "presentViewController:animated:"
+      MethodKind: Instance
+      SwiftName: "present(_:animated:)"
+
+    - Class: NSBundle
+      SwiftName: Bundle
+
+:Availability, AvailabilityMsg:
+
+  A value of "nonswift" is equivalent to ``NS_SWIFT_UNAVAILABLE``. A value of
+  "available" can be used in the "SwiftVersions" section to undo the effect of
+  "nonswift".
+
+  ::
+
+    - Selector: "dealloc"
+      MethodKind: Instance
+      Availability: nonswift
+      AvailabilityMsg: "prefer 'deinit'"
+
+:SwiftPrivate:
+
+  Equivalent to NS_REFINED_FOR_SWIFT.
+
+  ::
+
+    - Name: CGColorEqualToColor
+      SwiftPrivate: true
+
+:Nullability:
+
+  Used for properties and globals. There are four options, identified by their
+  initials:
+
+  - ``Nonnull`` or ``N`` (corresponding to ``_Nonnull``)
+  - ``Optional`` or ``O`` (corresponding to ``_Nullable``)
+  - ``Unspecified`` or ``U`` (corresponding to ``_Null_unspecified``)
+  - ``Scalar`` or ``S`` (deprecated)
+
+  Note that 'Nullability' is overridden by 'Type', even in a "SwiftVersions"
+  section.
+
+  .. note::
+
+    'Nullability' can also be used to describe the argument types of methods
+    and functions, but this usage is deprecated in favor of 'Parameters' (see
+    below).
+
+  ::
+
+    - Name: dataSource
+      Nullability: O
+
+:NullabilityOfRet:
+
+  Used for methods and functions. Describes the nullability of the return type.
+
+  Note that 'NullabilityOfRet' is overridden by 'ResultType', even in a
+  "SwiftVersions" section.
+
+  .. warning::
+
+    Due to a compiler bug, 'NullabilityOfRet' may change nullability of the
+    parameters as well (rdar://30544062). Avoid using it and instead use
+    'ResultType' and specify the return type along with a nullability
+    annotation (see documentation for 'ResultType').
+
+  ::
+
+    - Selector: superclass
+      MethodKind: Class
+      NullabilityOfRet: O
+
+:Type:
+
+  Used for properties and globals. This completely overrides the type of the
+  declaration; it should ideally only be used for Swift backwards
+  compatibility, when existing type information has been made more precise in a
+  header. Prefer 'Nullability' and other annotations when possible.
+
+  We parse the specified type as if it appeared at the location of the
+  declaration whose type is being modified.  Macros are not available and
+  nullability must be applied explicitly (even in an ``NS_ASSUME_NONNULL_BEGIN``
+  section).
+
+  ::
+
+    - Name: delegate
+      PropertyKind: Instance
+      Type: "id"
+
+:ResultType:
+
+  Used for methods and functions. This completely overrides the return type; it
+  should ideally only be used for Swift backwards compatibility, when existing
+  type information has been made more precise in a header.
+
+  We parse the specified type as if it appeared at the location of the
+  declaration whose type is being modified.  Macros are not available and
+  nullability must be applied explicitly (even in an ``NS_ASSUME_NONNULL_BEGIN``
+  section).
+
+  ::
+
+    - Selector: "subviews"
+      MethodKind: Instance
+      ResultType: "NSArray * _Nonnull"
+
+:SwiftImportAsAccessors:
+
+  Used for properties. If true, the property will be exposed in Swift as its
+  accessor methods, rather than as a computed property using ``var``.
+
+  ::
+
+    - Name: currentContext
+      PropertyKind: Class
+      SwiftImportAsAccessors: true
+
+:NSErrorDomain:
+
+  Used for ``NSError`` code enums. The value is the name of the associated
+  domain ``NSString`` constant; an empty string (``""``) means the enum is a
+  normal enum rather than an error code.
+
+  ::
+
+    - Name: MKErrorCode
+      NSErrorDomain: MKErrorDomain
+
+:SwiftWrapper:
+
+  Controls ``NS_STRING_ENUM`` and ``NS_EXTENSIBLE_STRING_ENUM``. There are three
+  options:
+
+  - "struct" (extensible)
+  - "enum"
+  - "none"
+
+  Note that even an "enum" wrapper is still presented as a struct in Swift;
+  it's just a "more enum-like" struct.
+
+  ::
+
+    - Name: AVMediaType
+      SwiftWrapper: none
+
+:EnumKind:
+
+  Has the same effect as ``NS_ENUM`` and ``NS_OPTIONS``. There are four options:
+
+  - "NSEnum" / "CFEnum"
+  - "NSClosedEnum" / "CFClosedEnum"
+  - "NSOptions" / "CFOptions"
+  - "none"
+
+  ::
+
+    - Name: GKPhotoSize
+      EnumKind: none
+
+:Parameters:
+
+  Used for methods and functions. Parameters are identified by a 0-based
+  'Position' and support the 'Nullability', 'NoEscape', and 'Type' keys.
+
+  .. note::
+
+    Using 'Parameters' within a parameter entry to describe the parameters of a
+    block is not implemented. Use 'Type' on the entire parameter instead.
+
+  ::
+
+    - Selector: "isEqual:"
+      MethodKind: Instance
+      Parameters:
+      - Position: 0
+        Nullability: O
+
+:NoEscape:
+
+  Used only for block parameters. Equivalent to ``NS_NOESCAPE``.
+
+  ::
+
+    - Name: dispatch_sync
+      Parameters:
+      - Position: 0
+        NoEscape: true
+
+:SwiftBridge:
+
+  Used for Objective-C class types bridged to Swift value types. An empty
+  string ("") means a type is not bridged. Not supported outside of Apple
+  frameworks (the Swift side of it requires conforming to implementation-detail
+  protocols that are subject to change).
+
+  ::
+
+    - Name: NSIndexSet
+      SwiftBridge: IndexSet
+
+:DesignatedInit:
+
+  Used for init methods. Equivalent to ``NS_DESIGNATED_INITIALIZER``.
+
+  ::
+
+    - Selector: "initWithFrame:"
+      MethodKind: Instance
+      DesignatedInit: true

clang/test/APINotes/Inputs/Frameworks/SomeKit.framework/Headers/SomeKit.h

+#ifndef SOMEKIT_H
+#define SOMEKIT_H
+
+__attribute__((objc_root_class))
+@interface A
+-(A*)transform:(A*)input;
+-(A*)transform:(A*)input integer:(int)integer;
+
+@property (nonatomic, readonly, retain) A* someA;
+@property (nonatomic, retain) A* someOtherA;
+
+@property (nonatomic) int intValue;
+@end
+
+@interface B : A
+@end
+
+@interface C : A
+- (instancetype)init;
+- (instancetype)initWithA:(A*)a;
+@end
+
+@interface ProcessInfo : A
++(instancetype)processInfo;
+@end
+
+@interface A(NonNullProperties)
+@property (nonatomic, readwrite, retain) A *nonnullAInstance;
+@property (class, nonatomic, readwrite, retain) A *nonnullAInstance;
+
+@property (nonatomic, readwrite, retain) A *nonnullAClass;
+@property (class, nonatomic, readwrite, retain) A *nonnullAClass;
+
+@property (nonatomic, readwrite, retain) A *nonnullABoth;
+@property (class, nonatomic, readwrite, retain) A *nonnullABoth;
+@end
+
+#import <SomeKit/SomeKitExplicitNullability.h>
+
+extern int *global_int_ptr;
+
+int *global_int_fun(int *ptr, int *ptr2);
+
+#define SOMEKIT_DOUBLE double
+
+__attribute__((objc_root_class))
+@interface OverriddenTypes
+-(int *)methodToMangle:(int *)ptr1 second:(int *)ptr2;
+@property int *intPropertyToMangle;
+@end
+
+@interface A(ImplicitGetterSetters)
+@property (nonatomic, readonly, retain) A *implicitGetOnlyInstance;
+@property (class, nonatomic, readonly, retain) A *implicitGetOnlyClass;
+
+@property (nonatomic, readwrite, retain) A *implicitGetSetInstance;
+@property (class, nonatomic, readwrite, retain) A *implicitGetSetClass;
+@end
+
+#endif

clang/test/APINotes/Inputs/Frameworks/SomeKit.framework/Headers/SomeKit.apinotes

+Name: SomeKit
+Classes:
+  - Name: A
+    Methods:
+      - Selector:        "transform:"
+        MethodKind:      Instance
+        Availability:    none
+        AvailabilityMsg: "anything but this"
+      - Selector: "transform:integer:"
+        MethodKind:      Instance
+        NullabilityOfRet: N
+        Nullability:      [ N, S ]
+      - Selector: "implicitGetOnlyInstance"
+        MethodKind:      Instance
+        Availability:    none
+        AvailabilityMsg: "getter gone"
+      - Selector: "implicitGetOnlyClass"
+        MethodKind:      Class
+        Availability:    none
+        AvailabilityMsg: "getter gone"
+      - Selector: "implicitGetSetInstance"
+        MethodKind:      Instance
+        Availability:    none
+        AvailabilityMsg: "getter gone"
+      - Selector: "implicitGetSetClass"
+        MethodKind:      Class
+        Availability:    none
+        AvailabilityMsg: "getter gone"
+      - Selector: "setImplicitGetSetInstance:"
+        MethodKind:      Instance
+        Availability:    none
+        AvailabilityMsg: "setter gone"
+      - Selector: "setImplicitGetSetClass:"
+        MethodKind:      Class
+        Availability:    none
+        AvailabilityMsg: "setter gone"
+    Properties:
+      - Name: intValue
+        PropertyKind:    Instance
+        Availability: none
+        AvailabilityMsg: "wouldn't work anyway"
+      - Name: nonnullAInstance
+        PropertyKind:    Instance
+        Nullability:     N
+      - Name: nonnullAClass
+        PropertyKind:    Class
+        Nullability:     N
+      - Name: nonnullABoth
+        Nullability:     N
+  - Name: B
+    Availability: none
+    AvailabilityMsg: "just don't"
+  - Name: C
+    Methods:
+      - Selector: "initWithA:"
+        MethodKind: Instance
+        DesignatedInit: true
+  - Name: OverriddenTypes
+    Methods:
+      - Selector: "methodToMangle:second:"
+        MethodKind: Instance
+        ResultType: 'char *'
+        Parameters:
+          - Position: 0
+            Type: 'SOMEKIT_DOUBLE *'
+          - Position: 1
+            Type: 'float *'
+    Properties:
+      - Name: intPropertyToMangle
+        PropertyKind: Instance
+        Type: 'double *'
+Functions:
+  - Name: global_int_fun
+    ResultType: 'char *'
+    Parameters:
+      - Position: 0
+        Type: 'double *'
+      - Position: 1
+        Type: 'float *'
+Globals:
+  - Name: global_int_ptr
+    Type: 'double (*)(int, int)'
+SwiftVersions:
+  - Version: 3.0
+    Classes:
+      - Name: A
+        Methods:
+          - Selector: "transform:integer:"
+            MethodKind:      Instance
+            NullabilityOfRet: O
+            Nullability:      [ O, S ]
+        Properties:
+          - Name: explicitNonnullInstance
+            PropertyKind:    Instance
+            Nullability:     O
+          - Name: explicitNullableInstance
+            PropertyKind:    Instance
+            Nullability:     N

clang/test/APINotes/Inputs/Frameworks/SomeKit.framework/Headers/SomeKitExplicitNullability.h

+@interface A(ExplicitNullabilityProperties)
+@property (nonatomic, readwrite, retain, nonnull) A *explicitNonnullInstance;
+@property (nonatomic, readwrite, retain, nullable) A *explicitNullableInstance;
+@end

clang/test/APINotes/Inputs/Frameworks/SomeKit.framework/Headers/SomeKit_private.apinotes

+Name: SomeKit
+Classes:
+  - Name: A
+    Methods:
+      - Selector: "privateTransform:input:"
+        MethodKind:      Instance
+        NullabilityOfRet: N
+        Nullability:      [ N, S ]
+    Properties:
+      - Name: internalProperty
+        Nullability: N
+Protocols:
+  - Name: InternalProtocol
+    Availability: none
+    AvailabilityMsg: "not for you"