Skip to main content

Class: abstract ImportingPlugin

Defined in: src/plugins/core/importing-plugin.ts:68

Base class for plugins that import preprints from external sources.

Remarks​

Provides common functionality for importing preprints:

  • Import service integration
  • Deduplication checking
  • Rate limiting helpers
  • Event emission for import lifecycle

Subclasses must implement:

  • source - The external source identifier
  • fetchPreprints() - Fetches preprints from the external source

Subclasses may override:

  • parseExternalId() - Extracts external ID from URL
  • normalizePreprint() - Normalizes fetched data

Example​

export class ArxivPlugin extends ImportingPlugin {
  readonly id = 'pub.chive.plugin.arxiv';
  readonly source: ImportSource = 'arxiv';

  async *fetchPreprints(options?: FetchOptions): AsyncIterable<ExternalPreprint> {
    // Fetch from arXiv OAI-PMH endpoint
    const records = await this.oaiPmhClient.listRecords(options);
    for (const record of records) {
      yield this.mapToExternalPreprint(record);
    }
  }
}

Since​

0.1.0

Extends​

Extended by​

Constructors​

new ImportingPlugin()​

new ImportingPlugin(): ImportingPlugin

Returns​

ImportingPlugin

Inherited from​

BasePlugin.constructor

Properties​

cache​

protected cache: ICacheProvider

Defined in: src/plugins/builtin/base-plugin.ts:88

Cache provider (available after initialization).

Inherited from​

BasePlugin.cache

context​

protected context: IPluginContext

Defined in: src/plugins/builtin/base-plugin.ts:78

Plugin context (available after initialization).

Inherited from​

BasePlugin.context

id​

abstract readonly id: string

Defined in: src/plugins/builtin/base-plugin.ts:68

Unique plugin identifier.

Remarks​

Must match manifest.id.

Inherited from​

BasePlugin.id

importService​

protected importService: IImportService

Defined in: src/plugins/core/importing-plugin.ts:83

Import service instance.

Remarks​

Set during initialization via dependency injection from context.

logger​

protected logger: ILogger

Defined in: src/plugins/builtin/base-plugin.ts:83

Logger instance (available after initialization).

Inherited from​

BasePlugin.logger

manifest​

abstract readonly manifest: IPluginManifest

Defined in: src/plugins/builtin/base-plugin.ts:73

Plugin manifest.

Inherited from​

BasePlugin.manifest

metrics​

protected metrics: IMetrics

Defined in: src/plugins/builtin/base-plugin.ts:93

Metrics provider (available after initialization).

Inherited from​

BasePlugin.metrics

rateLimitDelayMs​

protected rateLimitDelayMs: number = 1000

Defined in: src/plugins/core/importing-plugin.ts:97

Minimum delay between requests in milliseconds.

Remarks​

Override in subclass to set source-specific rate limit. Default: 1000ms (1 request per second)

source​

abstract readonly source: string

Defined in: src/plugins/core/importing-plugin.ts:75

External source identifier.

Remarks​

Used for tracking imports and preventing duplicates.

Methods​

buildPdfUrl()​

buildPdfUrl(externalId): null | string

Defined in: src/plugins/core/importing-plugin.ts:397

Builds the PDF URL for a preprint (if available).

Parameters​

externalId​

string

Source-specific identifier

Returns​

null | string

PDF URL or null if not available

Remarks​

Override in subclass for source-specific PDF URL building. Default implementation returns null.

buildPreprintUrl()​

abstract buildPreprintUrl(externalId): string

Defined in: src/plugins/core/importing-plugin.ts:385

Builds the canonical URL for a preprint.

Parameters​

externalId​

string

Source-specific identifier

Returns​

string

Full URL to the preprint

Remarks​

Override in subclass for source-specific URL building.

fetchPreprints()​

abstract fetchPreprints(options?): AsyncIterable<ExternalPreprint>

Defined in: src/plugins/core/importing-plugin.ts:145

Fetches preprints from the external source.

Parameters​

options?​

FetchOptions

Fetch options (pagination, filters)

Returns​

AsyncIterable<ExternalPreprint>

Async iterable of external preprints

Remarks​

Implementations should:

  • Respect rate limits using rateLimit()
  • Handle pagination via cursor
  • Catch and log errors for individual items

Example​

async *fetchPreprints(options?: FetchOptions): AsyncIterable<ExternalPreprint> {
  let cursor = options?.cursor;
  do {
    await this.rateLimit();
    const response = await this.fetchPage(cursor);
    for (const item of response.items) {
      yield this.mapToPreprint(item);
    }
    cursor = response.nextCursor;
  } while (cursor && (!options?.limit || count < options.limit));
}

getConfig()​

protected getConfig<T>(key): undefined | T

Defined in: src/plugins/builtin/base-plugin.ts:226

Gets a configuration value with type safety.

Type Parameters​

• T

Parameters​

key​

string

Configuration key

Returns​

undefined | T

Configuration value or undefined

Example​

const apiKey = this.getConfig<string>('apiKey');

Inherited from​

BasePlugin.getConfig

getExistingImport()​

getExistingImport(externalId): Promise<null | ImportedPreprint>

Defined in: src/plugins/core/importing-plugin.ts:167

Gets an existing import by external ID.

Parameters​

externalId​

string

Source-specific identifier

Returns​

Promise<null | ImportedPreprint>

Imported preprint or null if not found

getRequiredConfig()​

protected getRequiredConfig<T>(key): T

Defined in: src/plugins/builtin/base-plugin.ts:242

Gets a required configuration value.

Type Parameters​

• T

Parameters​

key​

string

Configuration key

Returns​

T

Configuration value

Throws​

Error if key not found

Example​

const apiKey = this.getRequiredConfig<string>('apiKey');

Inherited from​

BasePlugin.getRequiredConfig

getState()​

getState(): PluginState

Defined in: src/plugins/builtin/base-plugin.ts:162

Gets current plugin state.

Returns​

PluginState

Current lifecycle state

Inherited from​

BasePlugin.getState

importPreprint()​

importPreprint(preprint): Promise<ImportedPreprint>

Defined in: src/plugins/core/importing-plugin.ts:185

Imports a preprint into the AppView cache.

Parameters​

preprint​

ExternalPreprint

External preprint data

Returns​

Promise<ImportedPreprint>

Created or existing imported preprint

Remarks​

Performs deduplication and emits import.created event on success.

Throws​

Error if import service unavailable

initialize()​

initialize(context): Promise<void>

Defined in: src/plugins/core/importing-plugin.ts:107

Initializes the importing plugin.

Parameters​

context​

IPluginContext

Plugin context with injected dependencies

Returns​

Promise<void>

Remarks​

Retrieves import service from context and calls onInitialize().

Overrides​

BasePlugin.initialize

isImported()​

isImported(externalId): Promise<boolean>

Defined in: src/plugins/core/importing-plugin.ts:153

Checks if a preprint has already been imported.

Parameters​

externalId​

string

Source-specific identifier

Returns​

Promise<boolean>

True if already imported

onInitialize()​

abstract protected onInitialize(): Promise<void>

Defined in: src/plugins/builtin/base-plugin.ts:190

Override in subclass for initialization logic.

Returns​

Promise<void>

Remarks​

Called during initialize() after context is set up. Use this to:

  • Subscribe to event hooks
  • Initialize plugin state
  • Validate configuration

Example​

protected async onInitialize(): Promise<void> {
  // Subscribe to events
  this.context.eventBus.on('preprint.indexed', this.handleIndexed.bind(this));

  // Load cached state
  const state = await this.cache.get('state');
  if (state) {
    this.restoreState(state);
  }
}

Inherited from​

BasePlugin.onInitialize

onShutdown()​

protected onShutdown(): Promise<void>

Defined in: src/plugins/builtin/base-plugin.ts:211

Override in subclass for shutdown logic.

Returns​

Promise<void>

Remarks​

Called during shutdown(). Use this to:

  • Save state
  • Close connections
  • Clean up resources

Default implementation does nothing.

Example​

protected async onShutdown(): Promise<void> {
  await this.saveState();
  this.connection?.close();
}

Inherited from​

BasePlugin.onShutdown

parseExternalId()​

parseExternalId(url): null | string

Defined in: src/plugins/core/importing-plugin.ts:370

Parses external ID from a URL.

Parameters​

url​

string

URL to parse

Returns​

null | string

External ID or null if not parseable

Remarks​

Override in subclass for source-specific URL parsing. Default implementation returns null.

Example​

// ArXiv: https://arxiv.org/abs/2401.12345
parseExternalId('https://arxiv.org/abs/2401.12345'); // '2401.12345'

rateLimit()​

protected rateLimit(): Promise<void>

Defined in: src/plugins/core/importing-plugin.ts:342

Enforces rate limiting for external requests.

Returns​

Promise<void>

Remarks​

Call this before making external API/HTTP requests. Delays execution if needed to respect rate limit.

recordCounter()​

protected recordCounter(name, labels?, value?): void

Defined in: src/plugins/builtin/base-plugin.ts:257

Records a counter metric.

Parameters​

name​

string

Metric name

labels?​

Record<string, string>

Optional labels

value?​

number

Optional increment value (default 1)

Returns​

void

Inherited from​

BasePlugin.recordCounter

recordGauge()​

protected recordGauge(name, value, labels?): void

Defined in: src/plugins/builtin/base-plugin.ts:268

Records a gauge metric.

Parameters​

name​

string

Metric name

value​

number

Gauge value

labels?​

Record<string, string>

Optional labels

Returns​

void

Inherited from​

BasePlugin.recordGauge

runImportCycle()​

runImportCycle(options?): Promise<ImportCycleResult>

Defined in: src/plugins/core/importing-plugin.ts:285

Runs a full import cycle.

Parameters​

options?​

FetchOptions

Fetch options

Returns​

Promise<ImportCycleResult>

Import statistics

Remarks​

Fetches preprints and imports new ones, updating existing ones. Respects rate limits and handles errors gracefully.

shutdown()​

shutdown(): Promise<void>

Defined in: src/plugins/builtin/base-plugin.ts:142

Shuts down the plugin.

Returns​

Promise<void>

Remarks​

Calls onShutdown() and updates state. Subclasses should override onShutdown() instead of this method.

Inherited from​

BasePlugin.shutdown

startTimer()​

protected startTimer(name, labels?): () => void

Defined in: src/plugins/builtin/base-plugin.ts:286

Starts a timer for histogram metrics.

Parameters​

name​

string

Metric name

labels?​

Record<string, string>

Optional labels

Returns​

Function

Function to call when operation completes

Returns​

void

Example​

const endTimer = this.startTimer('api_request');
await this.fetchData();
endTimer(); // Records duration

Inherited from​

BasePlugin.startTimer

updateImport()​

updateImport(id, preprint): Promise<ImportedPreprint>

Defined in: src/plugins/core/importing-plugin.ts:243

Updates an existing import with new data.

Parameters​

id​

number

Internal import ID

preprint​

Partial<ExternalPreprint>

Updated preprint data

Returns​

Promise<ImportedPreprint>

Updated imported preprint