The castle that was the seat of the last Provençale king stood blinking whitely at the edge of the water. The river marking the old border between Provence and France is not an average waterway. ‘Can I swim in the river?’ I ask the nice black-haired woman in tourist information. ‘That’s not a river!’ she replies, horrified. ‘That’s the Rhone.’

The city walls of Tarascon surround a net full of limestone cobbled streets and old shuttered mansions, jumbled like oysters. Despite being so pretty, so close to Arles and Avignon, there are almost no tourists here. Instead, there are cafés under plane trees, motorbike garages, bulls in the streets, a muralled cinema, and music. There also happens to be international-level contemporary art inside the fairytale castle: Marseille and the whole region of Provence is the 2013 European Capital of Culture.

The exhibition, Rives Imaginaires (Imaginary Shores), takes Ulysses as its inspiration. Provence has a history of wanderers and exiles. Martha and Mary Magdalene fleeing Palestine were the most famous of many migrants who crossed the Mediterranean, sailing up the Rhone, carrying stories and ideas to Tarascon.

The whole exhibition experience leaves me feeling rinsed in magic. I soak up the peace of the castle garden, lazing by a lotus trough with my book.

There’s a lovely Provençale word, souleiado – for the sun that breaks through storm-clouds. Souleiado is also the name of a famous fabric and fashion house started in Tarascon several hundred years ago. The old factory is now a museum, which I visit. Sun is streaming through the stained glass windows into the printing room where men hand-blocked their designs with mallets back in the 19th century; an exhibition downstairs showed fashion collections from 1950 to 1990. The colourful designs combined the Indian patterns imported to Marseille – fan and peacock motifs; boteh, lotus, chintz – with elements of Provençale folk art and the Camargue’s gypsy traditions: floral embroidery, polka dots, ruffles. In the shop, you can buy Souleiado’s full skirts and bra tops, 50s-style bikini sets, handkerchiefs, and men’s shirts with bull’s head motifs. The cloth is still printed locally, though now by roller, in Lyon.

I stay at the Hotel de Provence, a gorgeous 18th century hotel particulier built into the medieval city’s wall. My room is an oasis of calm in eau-de-Nil, where I can sit and write while watching the pigeons scurry to and fro across terracotta rooftops. I enjoy the sociable breakfasts around a single long table: excellent coffee, cheese and croissants in the handsome shady dining room.

The best place for dinner in Tarascon is Ariane Guilmain’s Restaurant Le Theatre – the tables are set out on the Renaissance terrace of the beautiful theatre. I eat trout with Camargue rice, washed down with local wine, and a soft warm slab of salted chocolate cake. At the Bistrot des Anges in the old market square, I lay into nectarine and cucumber salads at a brightly painted metal table in the midday sun. For late-night drinks, there’s Le T, a good bar-pizzeria.

A ten minute walk across the bridge takes you to Beaucaire, on the Rhone’s other bank. Beaucaire, an equally historic town, is livelier by night than Tarascon. Restaurants with music at outdoor stages line the quayside of the marina on the canal. A late night market springs to life here every Friday and Saturday in July and August, with stalls of jam and beer alongside suitcases of old dolls and tables of 1960s ceramics and handmade jewellery. I almost buy a Mylène Farmer cassette for old times’ sake (my brief French schooldays), but plump instead for a vintage leather wallet. After joining the crowds at the bull running at Square Eyssette - the bull in question seemed admirably lackadaisical – I slip off to a concert of Renaissance and contemporary theorbo music, by Paris musician Thibaut Roussel, in a medieval courtyard heavy with myrtle blossom.

The moon is full, almost bulging out of itself, a flooding navel high above the masts by the time I leave to go back to Tarascon. The Rhone runs darkly under the bridge, carrying less than it once did, but still this place where people and ideas can sail against the current, into the heart of things.

Getting there: Ryanair flies to Marseille and Beziers from Edinburgh. You can reach Tarascon by bus or train visit bonjourlafrance.com for tickets.

Hotel de Provence: rooms from €59, hotel-provence-tarascon.com

Fact file

Other Highlights of Marseille-Provence 2013, European Capital of Culture:

September at Sea

Events include kayaking, film screenings, catamaran and scuba trips, and ragga with moules frites.
Marseille, until 7 Oct.

Le Grand Atelier du Midi

Major exhibition of Impressionist and Cubist works.
Marseille and Aix-en-Provence, until 13 Oct.

Les Grands Chemins d’Envies Rhonements

Contemporary art trails including installations by Scottish artist Rob Mulholland.
The Camargue, until 15 Oct.

Monet, Renoir … Chagall at the Carrieres de Lumieres

Sound and light installation /exhibition inside old bauxite quarries.
Baux-en-Provence, until 5 Jan.

Le Corbusier and Brutalism

Exhibition at the J1 Hangar in Marseille.
11 Oct–12 Jan 2014.

Fiesta des Minots

Kids’ festival featuring puppetry and silent movies across Provence.
27 Oct–8 Nov.

MuCEM

Marseille’s brand new Rudy Ricciotti-designed showpiece museum, and its Mediterranean culture exhibitions.